diff --git a/docs/faq.rst b/docs/faq.rst index b92d173dd..f593e4f16 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -64,7 +64,7 @@ Does Scrapy work with HTTP proxies? Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP Proxy downloader middleware. See -:class:`~scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware`. +:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`. How can I scrape an item with attributes in different pages? ------------------------------------------------------------ @@ -113,7 +113,7 @@ See previous question. Can I use Basic HTTP Authentication in my spiders? -------------------------------------------------- -Yes, see :class:`~scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware`. +Yes, see :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`. Why does Scrapy download pages in English instead of my native language? ------------------------------------------------------------------------ diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 3d24b29ad..be19bc44f 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -42,7 +42,7 @@ as its value. For example, if you want to disable the user-agent middleware:: DOWNLOADER_MIDDLEWARES = { 'myproject.middlewares.CustomDownloaderMiddleware': 543, - 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None, + 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, } Finally, keep in mind that some middlewares may need to be enabled through a @@ -54,7 +54,7 @@ Writing your own downloader middleware Each middleware component is a Python class that defines one or more of the following methods: -.. module:: scrapy.contrib.downloadermiddleware +.. module:: scrapy.downloadermiddlewares .. class:: DownloaderMiddleware @@ -169,7 +169,7 @@ For a list of the components enabled by default (and their orders) see the CookiesMiddleware ----------------- -.. module:: scrapy.contrib.downloadermiddleware.cookies +.. module:: scrapy.downloadermiddlewares.cookies :synopsis: Cookies Downloader Middleware .. class:: CookiesMiddleware @@ -246,7 +246,7 @@ Here's an example of a log with :setting:`COOKIES_DEBUG` enabled:: DefaultHeadersMiddleware ------------------------ -.. module:: scrapy.contrib.downloadermiddleware.defaultheaders +.. module:: scrapy.downloadermiddlewares.defaultheaders :synopsis: Default Headers Downloader Middleware .. class:: DefaultHeadersMiddleware @@ -257,7 +257,7 @@ DefaultHeadersMiddleware DownloadTimeoutMiddleware ------------------------- -.. module:: scrapy.contrib.downloadermiddleware.downloadtimeout +.. module:: scrapy.downloadermiddlewares.downloadtimeout :synopsis: Download timeout middleware .. class:: DownloadTimeoutMiddleware @@ -275,7 +275,7 @@ DownloadTimeoutMiddleware HttpAuthMiddleware ------------------ -.. module:: scrapy.contrib.downloadermiddleware.httpauth +.. module:: scrapy.downloadermiddlewares.httpauth :synopsis: HTTP Auth downloader middleware .. class:: HttpAuthMiddleware @@ -304,7 +304,7 @@ HttpAuthMiddleware HttpCacheMiddleware ------------------- -.. module:: scrapy.contrib.downloadermiddleware.httpcache +.. module:: scrapy.downloadermiddlewares.httpcache :synopsis: HTTP Cache downloader middleware .. class:: HttpCacheMiddleware @@ -579,7 +579,7 @@ This setting is specific to the Filesystem backend. HttpCompressionMiddleware ------------------------- -.. module:: scrapy.contrib.downloadermiddleware.httpcompression +.. module:: scrapy.downloadermiddlewares.httpcompression :synopsis: Http Compression Middleware .. class:: HttpCompressionMiddleware @@ -603,7 +603,7 @@ Whether the Compression middleware will be enabled. ChunkedTransferMiddleware ------------------------- -.. module:: scrapy.contrib.downloadermiddleware.chunked +.. module:: scrapy.downloadermiddlewares.chunked :synopsis: Chunked Transfer Middleware .. class:: ChunkedTransferMiddleware @@ -613,7 +613,7 @@ ChunkedTransferMiddleware HttpProxyMiddleware ------------------- -.. module:: scrapy.contrib.downloadermiddleware.httpproxy +.. module:: scrapy.downloadermiddlewares.httpproxy :synopsis: Http Proxy Middleware .. versionadded:: 0.8 @@ -641,7 +641,7 @@ HttpProxyMiddleware RedirectMiddleware ------------------ -.. module:: scrapy.contrib.downloadermiddleware.redirect +.. module:: scrapy.downloadermiddlewares.redirect :synopsis: Redirection Middleware .. class:: RedirectMiddleware @@ -731,7 +731,7 @@ The maximum meta-refresh delay (in seconds) to follow the redirection. RetryMiddleware --------------- -.. module:: scrapy.contrib.downloadermiddleware.retry +.. module:: scrapy.downloadermiddlewares.retry :synopsis: Retry Middleware .. class:: RetryMiddleware @@ -800,7 +800,7 @@ connections lost, etc) are always retried. RobotsTxtMiddleware ------------------- -.. module:: scrapy.contrib.downloadermiddleware.robotstxt +.. module:: scrapy.downloadermiddlewares.robotstxt :synopsis: robots.txt middleware .. class:: RobotsTxtMiddleware @@ -828,7 +828,7 @@ the request will be ignored by this middleware even if DownloaderStats --------------- -.. module:: scrapy.contrib.downloadermiddleware.stats +.. module:: scrapy.downloadermiddlewares.stats :synopsis: Downloader Stats Middleware .. class:: DownloaderStats @@ -842,7 +842,7 @@ DownloaderStats UserAgentMiddleware ------------------- -.. module:: scrapy.contrib.downloadermiddleware.useragent +.. module:: scrapy.downloadermiddlewares.useragent :synopsis: User Agent Middleware .. class:: UserAgentMiddleware @@ -857,7 +857,7 @@ UserAgentMiddleware AjaxCrawlMiddleware ------------------- -.. module:: scrapy.contrib.downloadermiddleware.ajaxcrawl +.. module:: scrapy.downloadermiddlewares.ajaxcrawl .. class:: AjaxCrawlMiddleware diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 3936c04b8..3e68ad925 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -239,7 +239,7 @@ Default:: } The default headers used for Scrapy HTTP Requests. They're populated in the -:class:`~scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware`. +:class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`. .. setting:: DEPTH_LIMIT @@ -335,20 +335,20 @@ DOWNLOADER_MIDDLEWARES_BASE Default:: { - 'scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware': 100, - 'scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware': 300, - 'scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware': 350, - 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400, - 'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 500, - 'scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware': 550, - 'scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware': 580, - 'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware': 590, - 'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600, - 'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware': 700, - 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 750, - 'scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware': 830, - 'scrapy.contrib.downloadermiddleware.stats.DownloaderStats': 850, - 'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900, + 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100, + 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300, + 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350, + 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 400, + 'scrapy.downloadermiddlewares.retry.RetryMiddleware': 500, + 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 550, + 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580, + 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590, + 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600, + 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700, + 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750, + 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware': 830, + 'scrapy.downloadermiddlewares.stats.DownloaderStats': 850, + 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900, } A dict containing the downloader middlewares enabled by default in Scrapy. You @@ -837,7 +837,7 @@ ROBOTSTXT_OBEY Default: ``False`` -Scope: ``scrapy.contrib.downloadermiddleware.robotstxt`` +Scope: ``scrapy.downloadermiddlewares.robotstxt`` If enabled, Scrapy will respect robots.txt policies. For more information see :ref:`topics-dlmw-robots` diff --git a/scrapy/contrib_exp/downloadermiddleware/decompression.py b/scrapy/contrib_exp/downloadermiddleware/decompression.py index 8b1d61b3d..1f8490587 100644 --- a/scrapy/contrib_exp/downloadermiddleware/decompression.py +++ b/scrapy/contrib_exp/downloadermiddleware/decompression.py @@ -1,7 +1,7 @@ import warnings from scrapy.exceptions import ScrapyDeprecationWarning warnings.warn("Module `scrapy.contrib_exp.downloadermiddleware.decompression` is deprecated, " - "use `scrapy.contrib.downloadermiddleware.decompression` instead", + "use `scrapy.downloadermiddlewares.decompression` instead", ScrapyDeprecationWarning, stacklevel=2) -from scrapy.contrib.downloadermiddleware.decompression import DecompressionMiddleware +from scrapy.downloadermiddlewares.decompression import DecompressionMiddleware diff --git a/scrapy/contrib/downloadermiddleware/__init__.py b/scrapy/downloadermiddlewares/__init__.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/__init__.py rename to scrapy/downloadermiddlewares/__init__.py diff --git a/scrapy/contrib/downloadermiddleware/ajaxcrawl.py b/scrapy/downloadermiddlewares/ajaxcrawl.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/ajaxcrawl.py rename to scrapy/downloadermiddlewares/ajaxcrawl.py diff --git a/scrapy/contrib/downloadermiddleware/chunked.py b/scrapy/downloadermiddlewares/chunked.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/chunked.py rename to scrapy/downloadermiddlewares/chunked.py diff --git a/scrapy/contrib/downloadermiddleware/cookies.py b/scrapy/downloadermiddlewares/cookies.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/cookies.py rename to scrapy/downloadermiddlewares/cookies.py diff --git a/scrapy/contrib/downloadermiddleware/decompression.py b/scrapy/downloadermiddlewares/decompression.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/decompression.py rename to scrapy/downloadermiddlewares/decompression.py diff --git a/scrapy/contrib/downloadermiddleware/defaultheaders.py b/scrapy/downloadermiddlewares/defaultheaders.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/defaultheaders.py rename to scrapy/downloadermiddlewares/defaultheaders.py diff --git a/scrapy/contrib/downloadermiddleware/downloadtimeout.py b/scrapy/downloadermiddlewares/downloadtimeout.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/downloadtimeout.py rename to scrapy/downloadermiddlewares/downloadtimeout.py diff --git a/scrapy/contrib/downloadermiddleware/httpauth.py b/scrapy/downloadermiddlewares/httpauth.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/httpauth.py rename to scrapy/downloadermiddlewares/httpauth.py diff --git a/scrapy/contrib/downloadermiddleware/httpcache.py b/scrapy/downloadermiddlewares/httpcache.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/httpcache.py rename to scrapy/downloadermiddlewares/httpcache.py diff --git a/scrapy/contrib/downloadermiddleware/httpcompression.py b/scrapy/downloadermiddlewares/httpcompression.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/httpcompression.py rename to scrapy/downloadermiddlewares/httpcompression.py diff --git a/scrapy/contrib/downloadermiddleware/httpproxy.py b/scrapy/downloadermiddlewares/httpproxy.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/httpproxy.py rename to scrapy/downloadermiddlewares/httpproxy.py diff --git a/scrapy/contrib/downloadermiddleware/redirect.py b/scrapy/downloadermiddlewares/redirect.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/redirect.py rename to scrapy/downloadermiddlewares/redirect.py diff --git a/scrapy/contrib/downloadermiddleware/retry.py b/scrapy/downloadermiddlewares/retry.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/retry.py rename to scrapy/downloadermiddlewares/retry.py diff --git a/scrapy/contrib/downloadermiddleware/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/robotstxt.py rename to scrapy/downloadermiddlewares/robotstxt.py diff --git a/scrapy/contrib/downloadermiddleware/stats.py b/scrapy/downloadermiddlewares/stats.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/stats.py rename to scrapy/downloadermiddlewares/stats.py diff --git a/scrapy/contrib/downloadermiddleware/useragent.py b/scrapy/downloadermiddlewares/useragent.py similarity index 100% rename from scrapy/contrib/downloadermiddleware/useragent.py rename to scrapy/downloadermiddlewares/useragent.py diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 0c3d7c5bd..3323386aa 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -80,21 +80,21 @@ DOWNLOADER_MIDDLEWARES = {} DOWNLOADER_MIDDLEWARES_BASE = { # Engine side - 'scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware': 100, - 'scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware': 300, - 'scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware': 350, - 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400, - 'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 500, - 'scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware': 550, - 'scrapy.contrib.downloadermiddleware.ajaxcrawl.AjaxCrawlMiddleware': 560, - 'scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware': 580, - 'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware': 590, - 'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600, - 'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware': 700, - 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 750, - 'scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware': 830, - 'scrapy.contrib.downloadermiddleware.stats.DownloaderStats': 850, - 'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900, + 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100, + 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300, + 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350, + 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 400, + 'scrapy.downloadermiddlewares.retry.RetryMiddleware': 500, + 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 550, + 'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560, + 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580, + 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590, + 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600, + 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700, + 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750, + 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware': 830, + 'scrapy.downloadermiddlewares.stats.DownloaderStats': 850, + 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900, # Downloader side } diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py index 3152db6c7..f1aa4fcdb 100644 --- a/scrapy/utils/misc.py +++ b/scrapy/utils/misc.py @@ -32,7 +32,7 @@ def load_object(path): """Load an object given its absolute object path, and return it. object can be a class, function, variable o instance. - path ie: 'scrapy.contrib.downloadermiddelware.redirect.RedirectMiddleware' + path ie: 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware' """ try: diff --git a/tests/py3-ignores.txt b/tests/py3-ignores.txt index d27d5dd3c..cf814c15c 100644 --- a/tests/py3-ignores.txt +++ b/tests/py3-ignores.txt @@ -91,6 +91,9 @@ scrapy/contrib/linkextractors/htmlparser.py scrapy/contrib/downloadermiddleware/retry.py scrapy/contrib/downloadermiddleware/httpproxy.py scrapy/contrib/downloadermiddleware/cookies.py +scrapy/downloadermiddlewares/retry.py +scrapy/downloadermiddlewares/httpproxy.py +scrapy/downloadermiddlewares/cookies.py scrapy/contrib/statsmailer.py scrapy/contrib/memusage.py scrapy/commands/deploy.py diff --git a/tests/test_downloadermiddleware_ajaxcrawlable.py b/tests/test_downloadermiddleware_ajaxcrawlable.py index e73e62538..11de6e22d 100644 --- a/tests/test_downloadermiddleware_ajaxcrawlable.py +++ b/tests/test_downloadermiddleware_ajaxcrawlable.py @@ -1,11 +1,11 @@ import unittest -from scrapy.contrib.downloadermiddleware.ajaxcrawl import AjaxCrawlMiddleware +from scrapy.downloadermiddlewares.ajaxcrawl import AjaxCrawlMiddleware from scrapy.spider import Spider from scrapy.http import Request, HtmlResponse, Response from scrapy.utils.test import get_crawler -__doctests__ = ['scrapy.contrib.downloadermiddleware.ajaxcrawl'] +__doctests__ = ['scrapy.downloadermiddlewares.ajaxcrawl'] class AjaxCrawlMiddlewareTest(unittest.TestCase): def setUp(self): diff --git a/tests/test_downloadermiddleware_cookies.py b/tests/test_downloadermiddleware_cookies.py index 3b1853c82..7f491f258 100644 --- a/tests/test_downloadermiddleware_cookies.py +++ b/tests/test_downloadermiddleware_cookies.py @@ -3,7 +3,7 @@ import re from scrapy.http import Response, Request from scrapy.spider import Spider -from scrapy.contrib.downloadermiddleware.cookies import CookiesMiddleware +from scrapy.downloadermiddlewares.cookies import CookiesMiddleware class CookiesMiddlewareTest(TestCase): diff --git a/tests/test_downloadermiddleware_decompression.py b/tests/test_downloadermiddleware_decompression.py index f3f862604..7aca415ef 100644 --- a/tests/test_downloadermiddleware_decompression.py +++ b/tests/test_downloadermiddleware_decompression.py @@ -1,6 +1,6 @@ from unittest import TestCase, main from scrapy.http import Response, XmlResponse -from scrapy.contrib.downloadermiddleware.decompression import DecompressionMiddleware +from scrapy.downloadermiddlewares.decompression import DecompressionMiddleware from scrapy.spider import Spider from tests import get_testdata from scrapy.utils.test import assert_samelines @@ -16,7 +16,7 @@ def _test_data(formats): class DecompressionMiddlewareTest(TestCase): - + test_formats = ['tar', 'xml.bz2', 'xml.gz', 'zip'] uncompressed_body, test_responses = _test_data(test_formats) diff --git a/tests/test_downloadermiddleware_defaultheaders.py b/tests/test_downloadermiddleware_defaultheaders.py index 09973b367..26520a20a 100644 --- a/tests/test_downloadermiddleware_defaultheaders.py +++ b/tests/test_downloadermiddleware_defaultheaders.py @@ -1,7 +1,7 @@ from unittest import TestCase import six -from scrapy.contrib.downloadermiddleware.defaultheaders import DefaultHeadersMiddleware +from scrapy.downloadermiddlewares.defaultheaders import DefaultHeadersMiddleware from scrapy.http import Request from scrapy.spider import Spider from scrapy.utils.test import get_crawler diff --git a/tests/test_downloadermiddleware_downloadtimeout.py b/tests/test_downloadermiddleware_downloadtimeout.py index aba06686a..282d10829 100644 --- a/tests/test_downloadermiddleware_downloadtimeout.py +++ b/tests/test_downloadermiddleware_downloadtimeout.py @@ -1,6 +1,6 @@ import unittest -from scrapy.contrib.downloadermiddleware.downloadtimeout import DownloadTimeoutMiddleware +from scrapy.downloadermiddlewares.downloadtimeout import DownloadTimeoutMiddleware from scrapy.spider import Spider from scrapy.http import Request from scrapy.utils.test import get_crawler diff --git a/tests/test_downloadermiddleware_httpauth.py b/tests/test_downloadermiddleware_httpauth.py index adfcd802d..cef65b336 100644 --- a/tests/test_downloadermiddleware_httpauth.py +++ b/tests/test_downloadermiddleware_httpauth.py @@ -1,7 +1,7 @@ import unittest from scrapy.http import Request -from scrapy.contrib.downloadermiddleware.httpauth import HttpAuthMiddleware +from scrapy.downloadermiddlewares.httpauth import HttpAuthMiddleware from scrapy.spider import Spider class TestSpider(Spider): diff --git a/tests/test_downloadermiddleware_httpcache.py b/tests/test_downloadermiddleware_httpcache.py index 663d67be0..9c1678488 100644 --- a/tests/test_downloadermiddleware_httpcache.py +++ b/tests/test_downloadermiddleware_httpcache.py @@ -12,7 +12,7 @@ from scrapy.spider import Spider from scrapy.settings import Settings from scrapy.exceptions import IgnoreRequest from scrapy.utils.test import get_crawler -from scrapy.contrib.downloadermiddleware.httpcache import HttpCacheMiddleware +from scrapy.downloadermiddlewares.httpcache import HttpCacheMiddleware class _BaseTest(unittest.TestCase): diff --git a/tests/test_downloadermiddleware_httpcompression.py b/tests/test_downloadermiddleware_httpcompression.py index 1cc6f44c1..98df6d608 100644 --- a/tests/test_downloadermiddleware_httpcompression.py +++ b/tests/test_downloadermiddleware_httpcompression.py @@ -5,7 +5,7 @@ from gzip import GzipFile from scrapy.spider import Spider from scrapy.http import Response, Request, HtmlResponse -from scrapy.contrib.downloadermiddleware.httpcompression import HttpCompressionMiddleware +from scrapy.downloadermiddlewares.httpcompression import HttpCompressionMiddleware from tests import tests_datadir from w3lib.encoding import resolve_encoding diff --git a/tests/test_downloadermiddleware_httpproxy.py b/tests/test_downloadermiddleware_httpproxy.py index 58825c6cf..90609879c 100644 --- a/tests/test_downloadermiddleware_httpproxy.py +++ b/tests/test_downloadermiddleware_httpproxy.py @@ -2,7 +2,7 @@ import os import sys from twisted.trial.unittest import TestCase, SkipTest -from scrapy.contrib.downloadermiddleware.httpproxy import HttpProxyMiddleware +from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware from scrapy.exceptions import NotConfigured from scrapy.http import Response, Request from scrapy.spider import Spider diff --git a/tests/test_downloadermiddleware_redirect.py b/tests/test_downloadermiddleware_redirect.py index fb70b13ae..3f299f258 100644 --- a/tests/test_downloadermiddleware_redirect.py +++ b/tests/test_downloadermiddleware_redirect.py @@ -1,6 +1,6 @@ import unittest -from scrapy.contrib.downloadermiddleware.redirect import RedirectMiddleware, MetaRefreshMiddleware +from scrapy.downloadermiddlewares.redirect import RedirectMiddleware, MetaRefreshMiddleware from scrapy.spider import Spider from scrapy.exceptions import IgnoreRequest from scrapy.http import Request, Response, HtmlResponse diff --git a/tests/test_downloadermiddleware_retry.py b/tests/test_downloadermiddleware_retry.py index 669abea05..969452cfb 100644 --- a/tests/test_downloadermiddleware_retry.py +++ b/tests/test_downloadermiddleware_retry.py @@ -5,7 +5,7 @@ from twisted.internet.error import TimeoutError, DNSLookupError, \ ConnectionLost, TCPTimedOutError from scrapy import optional_features -from scrapy.contrib.downloadermiddleware.retry import RetryMiddleware +from scrapy.downloadermiddlewares.retry import RetryMiddleware from scrapy.xlib.tx import ResponseFailed from scrapy.spider import Spider from scrapy.http import Request, Response diff --git a/tests/test_downloadermiddleware_robotstxt.py b/tests/test_downloadermiddleware_robotstxt.py index 8b7ac7a6b..bc245dcec 100644 --- a/tests/test_downloadermiddleware_robotstxt.py +++ b/tests/test_downloadermiddleware_robotstxt.py @@ -3,7 +3,7 @@ import re from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.trial import unittest -from scrapy.contrib.downloadermiddleware.robotstxt import RobotsTxtMiddleware +from scrapy.downloadermiddlewares.robotstxt import RobotsTxtMiddleware from scrapy.exceptions import IgnoreRequest, NotConfigured from scrapy.http import Request, Response from scrapy.settings import Settings diff --git a/tests/test_downloadermiddleware_stats.py b/tests/test_downloadermiddleware_stats.py index b790ff09a..64f2d0786 100644 --- a/tests/test_downloadermiddleware_stats.py +++ b/tests/test_downloadermiddleware_stats.py @@ -1,6 +1,6 @@ from unittest import TestCase -from scrapy.contrib.downloadermiddleware.stats import DownloaderStats +from scrapy.downloadermiddlewares.stats import DownloaderStats from scrapy.http import Request, Response from scrapy.spider import Spider from scrapy.utils.test import get_crawler @@ -22,7 +22,7 @@ class TestDownloaderStats(TestCase): self.mw.process_request(self.req, self.spider) self.assertEqual(self.crawler.stats.get_value('downloader/request_count', \ spider=self.spider), 1) - + def test_process_response(self): self.mw.process_response(self.req, self.res, self.spider) self.assertEqual(self.crawler.stats.get_value('downloader/response_count', \ diff --git a/tests/test_downloadermiddleware_useragent.py b/tests/test_downloadermiddleware_useragent.py index 909d03ba5..ddbb8f3ac 100644 --- a/tests/test_downloadermiddleware_useragent.py +++ b/tests/test_downloadermiddleware_useragent.py @@ -2,7 +2,7 @@ from unittest import TestCase from scrapy.spider import Spider from scrapy.http import Request -from scrapy.contrib.downloadermiddleware.useragent import UserAgentMiddleware +from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware from scrapy.utils.test import get_crawler