RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1218 from Curita/move-base-to-packages 

Move base classes to their packages
This commit is contained in:
Daniel Graña 2015-05-14 18:28:17 -03:00
parent 65aa9ccc70 53fdaa3f70
commit 1195f906d6
84 changed files with 450 additions and 430 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
conftest.py
View File

@ -16,6 +16,9 @@ collect_ignore = [
"scrapy/squeue.py",
"scrapy/log.py",
"scrapy/dupefilter.py",
"scrapy/command.py",
"scrapy/linkextractor.py",
"scrapy/spider.py",
] + _py_files("scrapy/contrib") + _py_files("scrapy/contrib_exp")

14
docs/intro/tutorial.rst
View File

@ -95,18 +95,18 @@ domain (or group of domains).
They define an initial list of URLs to download, how to follow links, and how
to parse the contents of pages to extract :ref:`items <topics-items>`.
To create a Spider, you must subclass :class:`scrapy.Spider <scrapy.spider.Spider>` and
define some attributes:
To create a Spider, you must subclass :class:`scrapy.Spider
<scrapy.spiders.Spider>` and define some attributes:
* :attr:`~scrapy.spider.Spider.name`: identifies the Spider. It must be
* :attr:`~scrapy.spiders.Spider.name`: identifies the Spider. It must be
unique, that is, you can't set the same name for different Spiders.
* :attr:`~scrapy.spider.Spider.start_urls`: a list of URLs where the
* :attr:`~scrapy.spiders.Spider.start_urls`: a list of URLs where the
Spider will begin to crawl from. The first pages downloaded will be those
listed here. The subsequent URLs will be generated successively from data
contained in the start URLs.
* :meth:`~scrapy.spider.Spider.parse`: a method of the spider, which will
* :meth:`~scrapy.spiders.Spider.parse`: a method of the spider, which will
be called with the downloaded :class:`~scrapy.http.Response` object of each
start URL. The response is passed to the method as the first and only
argument.
@ -114,7 +114,7 @@ define some attributes:
This method is responsible for parsing the response data and extracting
scraped data (as scraped items) and more URLs to follow.
The :meth:`~scrapy.spider.Spider.parse` method is in charge of processing
The :meth:`~scrapy.spiders.Spider.parse` method is in charge of processing
the response and returning scraped data (as :class:`~scrapy.item.Item`
objects) and more URLs to follow (as :class:`~scrapy.http.Request` objects).
@ -178,7 +178,7 @@ them the ``parse`` method of the spider as their callback function.
These Requests are scheduled, then executed, and :class:`scrapy.http.Response`
objects are returned and then fed back to the spider, through the
:meth:`~scrapy.spider.Spider.parse` method.
:meth:`~scrapy.spiders.Spider.parse` method.
Extracting Items
----------------

2
docs/topics/api.rst
View File

@ -31,7 +31,7 @@ how you :ref:`configure the downloader middlewares
.. class:: Crawler(spidercls, settings)
The Crawler object must be instantiated with a
:class:`scrapy.spider.Spider` subclass and a
:class:`scrapy.spiders.Spider` subclass and a
:class:`scrapy.settings.Settings` object.
.. attribute:: settings

6
docs/topics/downloader-middleware.rst
View File

@ -91,7 +91,7 @@ more of the following methods:
:type request: :class:`~scrapy.http.Request` object
:param spider: the spider for which this request is intended
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. method:: process_response(request, response, spider)
@ -118,7 +118,7 @@ more of the following methods:
:type response: :class:`~scrapy.http.Response` object
:param spider: the spider for which this response is intended
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. method:: process_exception(request, exception, spider)
@ -149,7 +149,7 @@ more of the following methods:
:type exception: an ``Exception`` object
:param spider: the spider for which this request is intended
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. _topics-downloader-middleware-ref:

6
docs/topics/item-pipeline.rst
View File

@ -36,7 +36,7 @@ Each item pipeline component is a Python class that must implement the following
:type item: :class:`~scrapy.item.Item` object or a dict
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
Additionally, they may also implement the following methods:
@ -45,14 +45,14 @@ Additionally, they may also implement the following methods:
This method is called when the spider is opened.
:param spider: the spider which was opened
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. method:: close_spider(self, spider)
This method is called when the spider is closed.
:param spider: the spider which was closed
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. method:: from_crawler(cls, crawler)

4
docs/topics/leaks.rst
View File

@ -92,7 +92,7 @@ subclasses):
* :class:`scrapy.http.Response`
* :class:`scrapy.item.Item`
* :class:`scrapy.selector.Selector`
* :class:`scrapy.spider.Spider`
* :class:`scrapy.spiders.Spider`
A real example
--------------
@ -155,7 +155,7 @@ For this reason, that function has a ``ignore`` argument which can be used to
ignore a particular class (and all its subclases). For
example, this won't show any live references to spiders::
>>> from scrapy.spider import Spider
>>> from scrapy.spiders import Spider
>>> prefs(ignore=Spider)
.. module:: scrapy.utils.trackref

6
docs/topics/link-extractors.rst
View File

@ -78,8 +78,8 @@ LxmlLinkExtractor
:param deny_extensions: a single value or list of strings containing
extensions that should be ignored when extracting links.
If not given, it will default to the
``IGNORED_EXTENSIONS`` list defined in the `scrapy.linkextractor`_
module.
``IGNORED_EXTENSIONS`` list defined in the
`scrapy.linkextractors`_ module.
:type deny_extensions: list
:param restrict_xpaths: is an XPath (or list of XPath's) which defines
@ -132,4 +132,4 @@ LxmlLinkExtractor
:type process_value: callable
.. _scrapy.linkextractor: https://github.com/scrapy/scrapy/blob/master/scrapy/linkextractor.py
.. _scrapy.linkextractors: https://github.com/scrapy/scrapy/blob/master/scrapy/linkextractors/__init__.py

2
docs/topics/logging.rst
View File

@ -94,7 +94,7 @@ path::
Logging from Spiders
====================
Scrapy provides a :data:`~scrapy.spider.Spider.logger` within each Spider
Scrapy provides a :data:`~scrapy.spiders.Spider.logger` within each Spider
instance, that can be accessed and used like this::
import scrapy

2
docs/topics/request-response.rst
View File

@ -37,7 +37,7 @@ Request objects
request (once its downloaded) as its first parameter. For more information
see :ref:`topics-request-response-ref-request-callback-arguments` below.
If a Request doesn't specify a callback, the spider's
:meth:`~scrapy.spider.Spider.parse` method will be used.
:meth:`~scrapy.spiders.Spider.parse` method will be used.
Note that if exceptions are raised during processing, errback is called instead.
:type callback: callable

2
docs/topics/settings.rst
View File

@ -67,7 +67,7 @@ Example::
Spiders (See the :ref:`topics-spiders` chapter for reference) can define their
own settings that will take precedence and override the project ones. They can
do so by setting their :attr:`scrapy.spider.Spider.custom_settings` attribute.
do so by setting their :attr:`scrapy.spiders.Spider.custom_settings` attribute.
3. Project settings module
--------------------------

2
docs/topics/shell.rst
View File

@ -74,7 +74,7 @@ Those objects are:
* ``crawler`` - the current :class:`~scrapy.crawler.Crawler` object.
* ``spider`` - the Spider which is known to handle the URL, or a
:class:`~scrapy.spider.Spider` object if there is no spider found for
:class:`~scrapy.spiders.Spider` object if there is no spider found for
the current URL
* ``request`` - a :class:`~scrapy.http.Request` object of the last fetched

20
docs/topics/signals.rst
View File

@ -74,7 +74,7 @@ item_scraped
:type item: dict or :class:`~scrapy.item.Item` object
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
:param response: the response from where the item was scraped
:type response: :class:`~scrapy.http.Response` object
@ -94,7 +94,7 @@ item_dropped
:type item: dict or :class:`~scrapy.item.Item` object
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
:param response: the response from where the item was dropped
:type response: :class:`~scrapy.http.Response` object
@ -116,7 +116,7 @@ spider_closed
This signal supports returning deferreds from their handlers.
:param spider: the spider which has been closed
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
:param reason: a string which describes the reason why the spider was closed. If
it was closed because the spider has completed scraping, the reason
@ -140,7 +140,7 @@ spider_opened
This signal supports returning deferreds from their handlers.
:param spider: the spider which has been opened
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
spider_idle
-----------
@ -164,7 +164,7 @@ spider_idle
This signal does not support returning deferreds from their handlers.
:param spider: the spider which has gone idle
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
spider_error
------------
@ -181,7 +181,7 @@ spider_error
:type response: :class:`~scrapy.http.Response` object
:param spider: the spider which raised the exception
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
request_scheduled
-----------------
@ -198,7 +198,7 @@ request_scheduled
:type request: :class:`~scrapy.http.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
request_dropped
-----------------
@ -215,7 +215,7 @@ request_dropped
:type request: :class:`~scrapy.http.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
response_received
-----------------
@ -235,7 +235,7 @@ response_received
:type request: :class:`~scrapy.http.Request` object
:param spider: the spider for which the response is intended
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
response_downloaded
-------------------
@ -254,6 +254,6 @@ response_downloaded
:type request: :class:`~scrapy.http.Request` object
:param spider: the spider for which the response is intended
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. _Failure: http://twistedmatrix.com/documents/current/api/twisted.python.failure.Failure.html

12
docs/topics/spider-middleware.rst
View File

@ -81,7 +81,7 @@ following methods:
:type response: :class:`~scrapy.http.Response` object
:param spider: the spider for which this response is intended
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. method:: process_spider_output(response, result, spider)
@ -102,7 +102,7 @@ following methods:
or :class:`~scrapy.item.Item` objects
:param spider: the spider whose result is being processed
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. method:: process_spider_exception(response, exception, spider)
@ -130,7 +130,7 @@ following methods:
:type exception: `Exception`_ object
:param spider: the spider which raised the exception
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. method:: process_start_requests(start_requests, spider)
@ -157,7 +157,7 @@ following methods:
:type start_requests: an iterable of :class:`~scrapy.http.Request`
:param spider: the spider to whom the start requests belong
:type spider: :class:`~scrapy.spider.Spider` object
:type spider: :class:`~scrapy.spiders.Spider` object
.. _Exception: https://docs.python.org/2/library/exceptions.html#exceptions.Exception
@ -272,7 +272,7 @@ OffsiteMiddleware
Filters out Requests for URLs outside the domains covered by the spider.
This middleware filters out every request whose host names aren't in the
spider's :attr:`~scrapy.spider.Spider.allowed_domains` attribute.
spider's :attr:`~scrapy.spiders.Spider.allowed_domains` attribute.
When your spider returns a request for a domain not belonging to those
covered by the spider, this middleware will log a debug message similar to
@ -287,7 +287,7 @@ OffsiteMiddleware
will be printed (but only for the first request filtered).
If the spider doesn't define an
:attr:`~scrapy.spider.Spider.allowed_domains` attribute, or the
:attr:`~scrapy.spiders.Spider.allowed_domains` attribute, or the
attribute is empty, the offsite middleware will allow all requests.
If the request has the :attr:`~scrapy.http.Request.dont_filter` attribute

11
docs/topics/spiders.rst
View File

@ -17,10 +17,10 @@ For spiders, the scraping cycle goes through something like this:
those requests.
The first requests to perform are obtained by calling the
:meth:`~scrapy.spider.Spider.start_requests` method which (by default)
:meth:`~scrapy.spiders.Spider.start_requests` method which (by default)
generates :class:`~scrapy.http.Request` for the URLs specified in the
:attr:`~scrapy.spider.Spider.start_urls` and the
:attr:`~scrapy.spider.Spider.parse` method as callback function for the
:attr:`~scrapy.spiders.Spider.start_urls` and the
:attr:`~scrapy.spiders.Spider.parse` method as callback function for the
Requests.
2. In the callback function, you parse the response (web page) and return either
@ -42,7 +42,7 @@ Even though this cycle applies (more or less) to any kind of spider, there are
different kinds of default spiders bundled into Scrapy for different purposes.
We will talk about those types here.
.. module:: scrapy.spider
.. module:: scrapy.spiders
:synopsis: Spiders base class, spider manager and spider middleware
.. _topics-spiders-ref:
@ -319,8 +319,7 @@ with a ``TestItem`` declared in a ``myproject.items`` module::
description = scrapy.Field()
.. module:: scrapy.spiders
:synopsis: Collection of generic spiders
.. currentmodule:: scrapy.spiders
CrawlSpider
-----------

2
extras/qpsclient.py
View File

@ -7,7 +7,7 @@ usage:
"""
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request

2
scrapy/__init__.py
View File

@ -45,7 +45,7 @@ if twisted_version >= (11, 1, 0):
optional_features.add('http11')
# Declare top-level shortcuts
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request, FormRequest
from scrapy.selector import Selector
from scrapy.item import Item, Field

2
scrapy/cmdline.py
View File

@ -8,7 +8,7 @@ import pkg_resources
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.xlib import lsprofcalltree
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.exceptions import UsageError
from scrapy.utils.misc import walk_modules
from scrapy.utils.project import inside_project, get_project_settings

112
scrapy/command.py
View File

@ -1,107 +1,7 @@
"""
Base class for Scrapy commands
"""
import os
from optparse import OptionGroup
from twisted.python import failure
import warnings
from scrapy.exceptions import ScrapyDeprecationWarning
warnings.warn("Module `scrapy.command` is deprecated, "
"use `scrapy.commands` instead",
ScrapyDeprecationWarning, stacklevel=2)
from scrapy.utils.conf import arglist_to_dict
from scrapy.exceptions import UsageError
class ScrapyCommand(object):
requires_project = False
crawler_process = None
# default settings to be used for this command instead of global defaults
default_settings = {}
exitcode = 0
def __init__(self):
self.settings = None # set in scrapy.cmdline
def set_crawler(self, crawler):
assert not hasattr(self, '_crawler'), "crawler already set"
self._crawler = crawler
def syntax(self):
"""
Command syntax (preferably one-line). Do not include command name.
"""
return ""
def short_desc(self):
"""
A short description of the command
"""
return ""
def long_desc(self):
"""A long description of the command. Return short description when not
available. It cannot contain newlines, since contents will be formatted
by optparser which removes newlines and wraps text.
"""
return self.short_desc()
def help(self):
"""An extensive help for the command. It will be shown when using the
"help" command. It can contain newlines, since not post-formatting will
be applied to its contents.
"""
return self.long_desc()
def add_options(self, parser):
"""
Populate option parse with options available for this command
"""
group = OptionGroup(parser, "Global Options")
group.add_option("--logfile", metavar="FILE",
help="log file. if omitted stderr will be used")
group.add_option("-L", "--loglevel", metavar="LEVEL", default=None,
help="log level (default: %s)" % self.settings['LOG_LEVEL'])
group.add_option("--nolog", action="store_true",
help="disable logging completely")
group.add_option("--profile", metavar="FILE", default=None,
help="write python cProfile stats to FILE")
group.add_option("--lsprof", metavar="FILE", default=None,
help="write lsprof profiling stats to FILE")
group.add_option("--pidfile", metavar="FILE",
help="write process ID to FILE")
group.add_option("-s", "--set", action="append", default=[], metavar="NAME=VALUE",
help="set/override setting (may be repeated)")
group.add_option("--pdb", action="store_true", help="enable pdb on failure")
parser.add_option_group(group)
def process_options(self, args, opts):
try:
self.settings.setdict(arglist_to_dict(opts.set),
priority='cmdline')
except ValueError:
raise UsageError("Invalid -s value, use -s NAME=VALUE", print_help=False)
if opts.logfile:
self.settings.set('LOG_ENABLED', True, priority='cmdline')
self.settings.set('LOG_FILE', opts.logfile, priority='cmdline')
if opts.loglevel:
self.settings.set('LOG_ENABLED', True, priority='cmdline')
self.settings.set('LOG_LEVEL', opts.loglevel, priority='cmdline')
if opts.nolog:
self.settings.set('LOG_ENABLED', False, priority='cmdline')
if opts.pidfile:
with open(opts.pidfile, "w") as f:
f.write(str(os.getpid()) + os.linesep)
if opts.pdb:
failure.startDebugMode()
def run(self, args, opts):
"""
Entry point for running commands
"""
raise NotImplementedError
from scrapy.commands import *

107
scrapy/commands/__init__.py
View File

@ -0,0 +1,107 @@
"""
Base class for Scrapy commands
"""
import os
from optparse import OptionGroup
from twisted.python import failure
from scrapy.utils.conf import arglist_to_dict
from scrapy.exceptions import UsageError
class ScrapyCommand(object):
requires_project = False
crawler_process = None
# default settings to be used for this command instead of global defaults
default_settings = {}
exitcode = 0
def __init__(self):
self.settings = None # set in scrapy.cmdline
def set_crawler(self, crawler):
assert not hasattr(self, '_crawler'), "crawler already set"
self._crawler = crawler
def syntax(self):
"""
Command syntax (preferably one-line). Do not include command name.
"""
return ""
def short_desc(self):
"""
A short description of the command
"""
return ""
def long_desc(self):
"""A long description of the command. Return short description when not
available. It cannot contain newlines, since contents will be formatted
by optparser which removes newlines and wraps text.
"""
return self.short_desc()
def help(self):
"""An extensive help for the command. It will be shown when using the
"help" command. It can contain newlines, since not post-formatting will
be applied to its contents.
"""
return self.long_desc()
def add_options(self, parser):
"""
Populate option parse with options available for this command
"""
group = OptionGroup(parser, "Global Options")
group.add_option("--logfile", metavar="FILE",
help="log file. if omitted stderr will be used")
group.add_option("-L", "--loglevel", metavar="LEVEL", default=None,
help="log level (default: %s)" % self.settings['LOG_LEVEL'])
group.add_option("--nolog", action="store_true",
help="disable logging completely")
group.add_option("--profile", metavar="FILE", default=None,
help="write python cProfile stats to FILE")
group.add_option("--lsprof", metavar="FILE", default=None,
help="write lsprof profiling stats to FILE")
group.add_option("--pidfile", metavar="FILE",
help="write process ID to FILE")
group.add_option("-s", "--set", action="append", default=[], metavar="NAME=VALUE",
help="set/override setting (may be repeated)")
group.add_option("--pdb", action="store_true", help="enable pdb on failure")
parser.add_option_group(group)
def process_options(self, args, opts):
try:
self.settings.setdict(arglist_to_dict(opts.set),
priority='cmdline')
except ValueError:
raise UsageError("Invalid -s value, use -s NAME=VALUE", print_help=False)
if opts.logfile:
self.settings.set('LOG_ENABLED', True, priority='cmdline')
self.settings.set('LOG_FILE', opts.logfile, priority='cmdline')
if opts.loglevel:
self.settings.set('LOG_ENABLED', True, priority='cmdline')
self.settings.set('LOG_LEVEL', opts.loglevel, priority='cmdline')
if opts.nolog:
self.settings.set('LOG_ENABLED', False, priority='cmdline')
if opts.pidfile:
with open(opts.pidfile, "w") as f:
f.write(str(os.getpid()) + os.linesep)
if opts.pdb:
failure.startDebugMode()
def run(self, args, opts):
"""
Entry point for running commands
"""
raise NotImplementedError

2
scrapy/commands/bench.py
View File

@ -5,7 +5,7 @@ import subprocess
from six.moves.urllib.parse import urlencode
import scrapy
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.linkextractors import LinkExtractor

2
scrapy/commands/check.py
View File

@ -4,7 +4,7 @@ import sys
from collections import defaultdict
from unittest import TextTestRunner, TextTestResult as _TextTestResult
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.contracts import ContractsManager
from scrapy.utils.misc import load_object
from scrapy.utils.conf import build_component_list

2
scrapy/commands/crawl.py
View File

@ -1,5 +1,5 @@
import os
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.utils.conf import arglist_to_dict
from scrapy.exceptions import UsageError

2
scrapy/commands/edit.py
View File

@ -1,6 +1,6 @@
import sys, os
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.exceptions import UsageError
class Command(ScrapyCommand):

2
scrapy/commands/fetch.py
View File

@ -1,7 +1,7 @@
from __future__ import print_function
from w3lib.url import is_url
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.http import Request
from scrapy.exceptions import UsageError
from scrapy.utils.spider import spidercls_for_request, DefaultSpider

2
scrapy/commands/genspider.py
View File

@ -7,7 +7,7 @@ from importlib import import_module
from os.path import join, dirname, abspath, exists, splitext
import scrapy
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.utils.template import render_templatefile, string_camelcase
from scrapy.exceptions import UsageError

2
scrapy/commands/list.py
View File

@ -1,5 +1,5 @@
from __future__ import print_function
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
class Command(ScrapyCommand):

2
scrapy/commands/parse.py
View File

@ -3,7 +3,7 @@ import logging
from w3lib.url import is_url
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.http import Request
from scrapy.item import BaseItem
from scrapy.utils import display

2
scrapy/commands/runspider.py
View File

@ -3,7 +3,7 @@ import os
from importlib import import_module
from scrapy.utils.spider import iter_spider_classes
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.exceptions import UsageError
from scrapy.utils.conf import arglist_to_dict

2
scrapy/commands/settings.py
View File

@ -1,5 +1,5 @@
from __future__ import print_function
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
class Command(ScrapyCommand):

2
scrapy/commands/shell.py
View File

@ -6,7 +6,7 @@ See documentation in docs/topics/shell.rst
from threading import Thread
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.shell import Shell
from scrapy.http import Request
from scrapy.utils.spider import spidercls_for_request, DefaultSpider

2
scrapy/commands/startproject.py
View File

@ -7,7 +7,7 @@ from os.path import join, exists, abspath
from shutil import copytree, ignore_patterns
import scrapy
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
from scrapy.utils.template import render_templatefile, string_camelcase
from scrapy.exceptions import UsageError

2
scrapy/commands/version.py
View File

@ -5,7 +5,7 @@ import platform
import twisted
import scrapy
from scrapy.command import ScrapyCommand
from scrapy.commands import ScrapyCommand
class Command(ScrapyCommand):

3
scrapy/commands/view.py
View File

@ -1,5 +1,4 @@
from scrapy.command import ScrapyCommand
from scrapy.commands import fetch
from scrapy.commands import fetch, ScrapyCommand
from scrapy.utils.response import open_in_browser
class Command(fetch.Command):

2
scrapy/crawler.py
View File

@ -138,7 +138,7 @@ class CrawlerRunner(object):
:param crawler_or_spidercls: already created crawler, or a spider class
or spider's name inside the project to create it
:type crawler_or_spidercls: :class:`~scrapy.crawler.Crawler` instance,
:class:`~scrapy.spider.Spider` subclass or string
:class:`~scrapy.spiders.Spider` subclass or string
:param list args: arguments to initialize the spider

105
scrapy/linkextractor.py
View File

@ -1,100 +1,7 @@
"""
Common code and definitions used by Link extractors (located in
scrapy.linkextractors).
"""
import re
from six.moves.urllib.parse import urlparse
import warnings
from scrapy.exceptions import ScrapyDeprecationWarning
warnings.warn("Module `scrapy.linkextractor` is deprecated, "
"use `scrapy.linkextractors` instead",
ScrapyDeprecationWarning, stacklevel=2)
from scrapy.selector.csstranslator import ScrapyHTMLTranslator
from scrapy.utils.url import url_is_from_any_domain
from scrapy.utils.url import canonicalize_url, url_is_from_any_domain, url_has_any_extension
from scrapy.utils.misc import arg_to_iter
# common file extensions that are not followed if they occur in links
IGNORED_EXTENSIONS = [
# images
'mng', 'pct', 'bmp', 'gif', 'jpg', 'jpeg', 'png', 'pst', 'psp', 'tif',
'tiff', 'ai', 'drw', 'dxf', 'eps', 'ps', 'svg',
# audio
'mp3', 'wma', 'ogg', 'wav', 'ra', 'aac', 'mid', 'au', 'aiff',
# video
'3gp', 'asf', 'asx', 'avi', 'mov', 'mp4', 'mpg', 'qt', 'rm', 'swf', 'wmv',
'm4a',
# office suites
'xls', 'xlsx', 'ppt', 'pptx', 'doc', 'docx', 'odt', 'ods', 'odg', 'odp',
# other
'css', 'pdf', 'exe', 'bin', 'rss', 'zip', 'rar',
]
_re_type = type(re.compile("", 0))
_matches = lambda url, regexs: any((r.search(url) for r in regexs))
_is_valid_url = lambda url: url.split('://', 1)[0] in set(['http', 'https', 'file'])
class FilteringLinkExtractor(object):
_csstranslator = ScrapyHTMLTranslator()
def __init__(self, link_extractor, allow, deny, allow_domains, deny_domains,
restrict_xpaths, canonicalize, deny_extensions, restrict_css):
self.link_extractor = link_extractor
self.allow_res = [x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(allow)]
self.deny_res = [x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(deny)]
self.allow_domains = set(arg_to_iter(allow_domains))
self.deny_domains = set(arg_to_iter(deny_domains))
self.restrict_xpaths = tuple(arg_to_iter(restrict_xpaths))
self.restrict_xpaths += tuple(map(self._csstranslator.css_to_xpath,
arg_to_iter(restrict_css)))
self.canonicalize = canonicalize
if deny_extensions is None:
deny_extensions = IGNORED_EXTENSIONS
self.deny_extensions = set(['.' + e for e in arg_to_iter(deny_extensions)])
def _link_allowed(self, link):
if not _is_valid_url(link.url):
return False
if self.allow_res and not _matches(link.url, self.allow_res):
return False
if self.deny_res and _matches(link.url, self.deny_res):
return False
parsed_url = urlparse(link.url)
if self.allow_domains and not url_is_from_any_domain(parsed_url, self.allow_domains):
return False
if self.deny_domains and url_is_from_any_domain(parsed_url, self.deny_domains):
return False
if self.deny_extensions and url_has_any_extension(parsed_url, self.deny_extensions):
return False
return True
def matches(self, url):
if self.allow_domains and not url_is_from_any_domain(url, self.allow_domains):
return False
if self.deny_domains and url_is_from_any_domain(url, self.deny_domains):
return False
allowed = [regex.search(url) for regex in self.allow_res] if self.allow_res else [True]
denied = [regex.search(url) for regex in self.deny_res] if self.deny_res else []
return any(allowed) and not any(denied)
def _process_links(self, links):
links = [x for x in links if self._link_allowed(x)]
if self.canonicalize:
for link in links:
link.url = canonicalize_url(urlparse(link.url))
links = self.link_extractor._process_links(links)
return links
def _extract_links(self, *args, **kwargs):
return self.link_extractor._extract_links(*args, **kwargs)
from scrapy.linkextractors import *

98
scrapy/linkextractors/__init__.py
View File

@ -5,4 +5,102 @@ This package contains a collection of Link Extractors.
For more info see docs/topics/link-extractors.rst
"""
import re
from six.moves.urllib.parse import urlparse
from scrapy.selector.csstranslator import ScrapyHTMLTranslator
from scrapy.utils.url import url_is_from_any_domain
from scrapy.utils.url import canonicalize_url, url_is_from_any_domain, url_has_any_extension
from scrapy.utils.misc import arg_to_iter
# common file extensions that are not followed if they occur in links
IGNORED_EXTENSIONS = [
# images
'mng', 'pct', 'bmp', 'gif', 'jpg', 'jpeg', 'png', 'pst', 'psp', 'tif',
'tiff', 'ai', 'drw', 'dxf', 'eps', 'ps', 'svg',
# audio
'mp3', 'wma', 'ogg', 'wav', 'ra', 'aac', 'mid', 'au', 'aiff',
# video
'3gp', 'asf', 'asx', 'avi', 'mov', 'mp4', 'mpg', 'qt', 'rm', 'swf', 'wmv',
'm4a',
# office suites
'xls', 'xlsx', 'ppt', 'pptx', 'doc', 'docx', 'odt', 'ods', 'odg', 'odp',
# other
'css', 'pdf', 'exe', 'bin', 'rss', 'zip', 'rar',
]
_re_type = type(re.compile("", 0))
_matches = lambda url, regexs: any((r.search(url) for r in regexs))
_is_valid_url = lambda url: url.split('://', 1)[0] in set(['http', 'https', 'file'])
class FilteringLinkExtractor(object):
_csstranslator = ScrapyHTMLTranslator()
def __init__(self, link_extractor, allow, deny, allow_domains, deny_domains,
restrict_xpaths, canonicalize, deny_extensions, restrict_css):
self.link_extractor = link_extractor
self.allow_res = [x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(allow)]
self.deny_res = [x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(deny)]
self.allow_domains = set(arg_to_iter(allow_domains))
self.deny_domains = set(arg_to_iter(deny_domains))
self.restrict_xpaths = tuple(arg_to_iter(restrict_xpaths))
self.restrict_xpaths += tuple(map(self._csstranslator.css_to_xpath,
arg_to_iter(restrict_css)))
self.canonicalize = canonicalize
if deny_extensions is None:
deny_extensions = IGNORED_EXTENSIONS
self.deny_extensions = set(['.' + e for e in arg_to_iter(deny_extensions)])
def _link_allowed(self, link):
if not _is_valid_url(link.url):
return False
if self.allow_res and not _matches(link.url, self.allow_res):
return False
if self.deny_res and _matches(link.url, self.deny_res):
return False
parsed_url = urlparse(link.url)
if self.allow_domains and not url_is_from_any_domain(parsed_url, self.allow_domains):
return False
if self.deny_domains and url_is_from_any_domain(parsed_url, self.deny_domains):
return False
if self.deny_extensions and url_has_any_extension(parsed_url, self.deny_extensions):
return False
return True
def matches(self, url):
if self.allow_domains and not url_is_from_any_domain(url, self.allow_domains):
return False
if self.deny_domains and url_is_from_any_domain(url, self.deny_domains):
return False
allowed = [regex.search(url) for regex in self.allow_res] if self.allow_res else [True]
denied = [regex.search(url) for regex in self.deny_res] if self.deny_res else []
return any(allowed) and not any(denied)
def _process_links(self, links):
links = [x for x in links if self._link_allowed(x)]
if self.canonicalize:
for link in links:
link.url = canonicalize_url(urlparse(link.url))
links = self.link_extractor._process_links(links)
return links
def _extract_links(self, *args, **kwargs):
return self.link_extractor._extract_links(*args, **kwargs)
# Top-level imports
from .lxmlhtml import LxmlLinkExtractor as LinkExtractor

2
scrapy/linkextractors/lxmlhtml.py
View File

@ -11,7 +11,7 @@ from scrapy.selector import Selector
from scrapy.link import Link
from scrapy.utils.misc import arg_to_iter
from scrapy.utils.python import unique as unique_list, str_to_unicode
from scrapy.linkextractor import FilteringLinkExtractor
from scrapy.linkextractors import FilteringLinkExtractor
from scrapy.utils.response import get_base_url

2
scrapy/linkextractors/sgml.py
View File

@ -8,7 +8,7 @@ from sgmllib import SGMLParser
from w3lib.url import safe_url_string
from scrapy.selector import Selector
from scrapy.link import Link
from scrapy.linkextractor import FilteringLinkExtractor
from scrapy.linkextractors import FilteringLinkExtractor
from scrapy.utils.misc import arg_to_iter
from scrapy.utils.python import unique as unique_list, str_to_unicode
from scrapy.utils.response import get_base_url

2
scrapy/shell.py
View File

@ -17,7 +17,7 @@ from scrapy.exceptions import IgnoreRequest, ScrapyDeprecationWarning
from scrapy.http import Request, Response
from scrapy.item import BaseItem
from scrapy.settings import Settings
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.utils.console import start_python_console
from scrapy.utils.misc import load_object
from scrapy.utils.response import open_in_browser

114
scrapy/spider.py
View File

@ -1,113 +1,7 @@
"""
Base class for Scrapy spiders
See documentation in docs/topics/spiders.rst
"""
import logging
import warnings
from scrapy import signals
from scrapy.http import Request
from scrapy.utils.trackref import object_ref
from scrapy.utils.url import url_is_from_spider
from scrapy.utils.deprecate import create_deprecated_class
from scrapy.exceptions import ScrapyDeprecationWarning
warnings.warn("Module `scrapy.spider` is deprecated, "
"use `scrapy.spiders` instead",
ScrapyDeprecationWarning, stacklevel=2)
class Spider(object_ref):
"""Base class for scrapy spiders. All spiders must inherit from this
class.
"""
name = None
custom_settings = None
def __init__(self, name=None, **kwargs):
if name is not None:
self.name = name
elif not getattr(self, 'name', None):
raise ValueError("%s must have a name" % type(self).__name__)
self.__dict__.update(kwargs)
if not hasattr(self, 'start_urls'):
self.start_urls = []
@property
def logger(self):
logger = logging.getLogger(self.name)
return logging.LoggerAdapter(logger, {'spider': self})
def log(self, message, level=logging.DEBUG, **kw):
"""Log the given message at the given log level
This helper wraps a log call to the logger within the spider, but you
can use it directly (e.g. Spider.logger.info('msg')) or use any other
Python logger too.
"""
self.logger.log(level, message, **kw)
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = cls(*args, **kwargs)
spider._set_crawler(crawler)
return spider
def set_crawler(self, crawler):
warnings.warn("set_crawler is deprecated, instantiate and bound the "
"spider to this crawler with from_crawler method "
"instead.",
category=ScrapyDeprecationWarning, stacklevel=2)
assert not hasattr(self, 'crawler'), "Spider already bounded to a " \
"crawler"
self._set_crawler(crawler)
def _set_crawler(self, crawler):
self.crawler = crawler
self.settings = crawler.settings
crawler.signals.connect(self.close, signals.spider_closed)
def start_requests(self):
for url in self.start_urls:
yield self.make_requests_from_url(url)
def make_requests_from_url(self, url):
return Request(url, dont_filter=True)
def parse(self, response):
raise NotImplementedError
@classmethod
def update_settings(cls, settings):
settings.setdict(cls.custom_settings or {}, priority='spider')
@classmethod
def handles_request(cls, request):
return url_is_from_spider(request.url, cls)
@staticmethod
def close(spider, reason):
closed = getattr(spider, 'closed', None)
if callable(closed):
return closed(reason)
def __str__(self):
return "<%s %r at 0x%0x>" % (type(self).__name__, self.name, id(self))
__repr__ = __str__
BaseSpider = create_deprecated_class('BaseSpider', Spider)
class ObsoleteClass(object):
def __init__(self, message):
self.message = message
def __getattr__(self, name):
raise AttributeError(self.message)
spiders = ObsoleteClass(
'"from scrapy.spider import spiders" no longer works - use '
'"from scrapy.spiderloader import SpiderLoader" and instantiate '
'it with your project settings"'
)
from scrapy.spiders import *

114
scrapy/spiders/__init__.py
View File

@ -1,3 +1,117 @@
"""
Base class for Scrapy spiders
See documentation in docs/topics/spiders.rst
"""
import logging
import warnings
from scrapy import signals
from scrapy.http import Request
from scrapy.utils.trackref import object_ref
from scrapy.utils.url import url_is_from_spider
from scrapy.utils.deprecate import create_deprecated_class
from scrapy.exceptions import ScrapyDeprecationWarning
class Spider(object_ref):
"""Base class for scrapy spiders. All spiders must inherit from this
class.
"""
name = None
custom_settings = None
def __init__(self, name=None, **kwargs):
if name is not None:
self.name = name
elif not getattr(self, 'name', None):
raise ValueError("%s must have a name" % type(self).__name__)
self.__dict__.update(kwargs)
if not hasattr(self, 'start_urls'):
self.start_urls = []
@property
def logger(self):
logger = logging.getLogger(self.name)
return logging.LoggerAdapter(logger, {'spider': self})
def log(self, message, level=logging.DEBUG, **kw):
"""Log the given message at the given log level
This helper wraps a log call to the logger within the spider, but you
can use it directly (e.g. Spider.logger.info('msg')) or use any other
Python logger too.
"""
self.logger.log(level, message, **kw)
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = cls(*args, **kwargs)
spider._set_crawler(crawler)
return spider
def set_crawler(self, crawler):
warnings.warn("set_crawler is deprecated, instantiate and bound the "
"spider to this crawler with from_crawler method "
"instead.",
category=ScrapyDeprecationWarning, stacklevel=2)
assert not hasattr(self, 'crawler'), "Spider already bounded to a " \
"crawler"
self._set_crawler(crawler)
def _set_crawler(self, crawler):
self.crawler = crawler
self.settings = crawler.settings
crawler.signals.connect(self.close, signals.spider_closed)
def start_requests(self):
for url in self.start_urls:
yield self.make_requests_from_url(url)
def make_requests_from_url(self, url):
return Request(url, dont_filter=True)
def parse(self, response):
raise NotImplementedError
@classmethod
def update_settings(cls, settings):
settings.setdict(cls.custom_settings or {}, priority='spider')
@classmethod
def handles_request(cls, request):
return url_is_from_spider(request.url, cls)
@staticmethod
def close(spider, reason):
closed = getattr(spider, 'closed', None)
if callable(closed):
return closed(reason)
def __str__(self):
return "<%s %r at 0x%0x>" % (type(self).__name__, self.name, id(self))
__repr__ = __str__
BaseSpider = create_deprecated_class('BaseSpider', Spider)
class ObsoleteClass(object):
def __init__(self, message):
self.message = message
def __getattr__(self, name):
raise AttributeError(self.message)
spiders = ObsoleteClass(
'"from scrapy.spider import spiders" no longer works - use '
'"from scrapy.spiderloader import SpiderLoader" and instantiate '
'it with your project settings"'
)
# Top-level imports
from scrapy.spiders.crawl import CrawlSpider, Rule
from scrapy.spiders.feed import XMLFeedSpider, CSVFeedSpider
from scrapy.spiders.sitemap import SitemapSpider

2
scrapy/spiders/crawl.py
View File

@ -9,7 +9,7 @@ import copy
from scrapy.http import Request, HtmlResponse
from scrapy.utils.spider import iterate_spider_output
from scrapy.spider import Spider
from scrapy.spiders import Spider
def identity(x):
return x

2
scrapy/spiders/feed.py
View File

@ -4,7 +4,7 @@ for scraping from an XML feed.
See documentation in docs/topics/spiders.rst
"""
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.utils.iterators import xmliter, csviter
from scrapy.utils.spider import iterate_spider_output
from scrapy.selector import Selector

6
scrapy/spiders/init.py
View File

@ -1,4 +1,4 @@
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.utils.spider import iterate_spider_output
class InitSpider(Spider):
@ -20,8 +20,8 @@ class InitSpider(Spider):
is called this spider is considered initialized. If you need to perform
several requests for initializing your spider, you can do so by using
different callbacks. The only requirement is that the final callback
(of the last initialization request) must be self.initialized.
(of the last initialization request) must be self.initialized.
The default implementation calls self.initialized immediately, and
means that no initialization is needed. This method should be
overridden only when you need to perform requests to initialize your

2
scrapy/spiders/sitemap.py
View File

@ -1,7 +1,7 @@
import re
import logging
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request, XmlResponse
from scrapy.utils.sitemap import Sitemap, sitemap_urls_from_robots
from scrapy.utils.gz import gunzip, is_gzipped

4
scrapy/utils/spider.py
View File

@ -3,7 +3,7 @@ import inspect
import six
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.utils.misc import arg_to_iter
logger = logging.getLogger(__name__)
@ -19,7 +19,7 @@ def iter_spider_classes(module):
"""
# this needs to be imported here until get rid of the spider manager
# singleton in scrapy.spider.spiders
from scrapy.spider import Spider
from scrapy.spiders import Spider
for obj in six.itervalues(vars(module)):
if inspect.isclass(obj) and \

2
scrapy/utils/test.py
View File

@ -27,7 +27,7 @@ def get_crawler(spidercls=None, settings_dict=None):
"""
from scrapy.crawler import CrawlerRunner
from scrapy.settings import Settings
from scrapy.spider import Spider
from scrapy.spiders import Spider
runner = CrawlerRunner(Settings(settings_dict))
return runner._create_crawler(spidercls or Spider)

2
tests/spiders.py
View File

@ -5,7 +5,7 @@ Some spiders used for testing and benchmarking
import time
from six.moves.urllib.parse import urlencode
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request
from scrapy.item import Item
from scrapy.linkextractors import LinkExtractor

2
tests/test_commands.py
View File

@ -158,7 +158,7 @@ class MySpider(scrapy.Spider):
fname = abspath(join(tmpdir, 'myspider.py'))
with open(fname, 'w') as f:
f.write("""
from scrapy.spider import Spider
from scrapy.spiders import Spider
""")
p = self.proc('runspider', fname)
log = p.stderr.read()

2
tests/test_contracts.py
View File

@ -2,7 +2,7 @@ from unittest import TextTestResult
from twisted.trial import unittest
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request
from scrapy.item import Item, Field
from scrapy.contracts import ContractsManager

2
tests/test_downloader_handlers.py
View File

@ -24,7 +24,7 @@ from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler
from scrapy.core.downloader.handlers.s3 import S3DownloadHandler
from scrapy.core.downloader.handlers.ftp import FTPDownloadHandler
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request
from scrapy.settings import Settings
from scrapy import optional_features

2
tests/test_downloadermiddleware.py
View File

@ -2,7 +2,7 @@ from twisted.trial.unittest import TestCase
from twisted.python.failure import Failure
from scrapy.http import Request, Response
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.core.downloader.middleware import DownloaderMiddlewareManager
from scrapy.utils.test import get_crawler

2
tests/test_downloadermiddleware_ajaxcrawlable.py
View File

@ -1,7 +1,7 @@
import unittest
from scrapy.downloadermiddlewares.ajaxcrawl import AjaxCrawlMiddleware
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request, HtmlResponse, Response
from scrapy.utils.test import get_crawler

2
tests/test_downloadermiddleware_cookies.py
View File

@ -2,7 +2,7 @@ from unittest import TestCase
import re
from scrapy.http import Response, Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.downloadermiddlewares.cookies import CookiesMiddleware

2
tests/test_downloadermiddleware_decompression.py
View File

@ -1,7 +1,7 @@
from unittest import TestCase, main
from scrapy.http import Response, XmlResponse
from scrapy.downloadermiddlewares.decompression import DecompressionMiddleware
from scrapy.spider import Spider
from scrapy.spiders import Spider
from tests import get_testdata
from scrapy.utils.test import assert_samelines

2
tests/test_downloadermiddleware_defaultheaders.py
View File

@ -3,7 +3,7 @@ import six
from scrapy.downloadermiddlewares.defaultheaders import DefaultHeadersMiddleware
from scrapy.http import Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.utils.test import get_crawler

2
tests/test_downloadermiddleware_downloadtimeout.py
View File

@ -1,7 +1,7 @@
import unittest
from scrapy.downloadermiddlewares.downloadtimeout import DownloadTimeoutMiddleware
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request
from scrapy.utils.test import get_crawler

2
tests/test_downloadermiddleware_httpauth.py
View File

@ -2,7 +2,7 @@ import unittest
from scrapy.http import Request
from scrapy.downloadermiddlewares.httpauth import HttpAuthMiddleware
from scrapy.spider import Spider
from scrapy.spiders import Spider
class TestSpider(Spider):
http_user = 'foo'

2
tests/test_downloadermiddleware_httpcache.py
View File

@ -8,7 +8,7 @@ from contextlib import contextmanager
import pytest
from scrapy.http import Response, HtmlResponse, Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.settings import Settings
from scrapy.exceptions import IgnoreRequest
from scrapy.utils.test import get_crawler

2
tests/test_downloadermiddleware_httpcompression.py
View File

@ -3,7 +3,7 @@ from unittest import TestCase
from os.path import join, abspath, dirname
from gzip import GzipFile
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Response, Request, HtmlResponse
from scrapy.downloadermiddlewares.httpcompression import HttpCompressionMiddleware
from tests import tests_datadir

2
tests/test_downloadermiddleware_httpproxy.py
View File

@ -5,7 +5,7 @@ from twisted.trial.unittest import TestCase, SkipTest
from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware
from scrapy.exceptions import NotConfigured
from scrapy.http import Response, Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
spider = Spider('foo')

2
tests/test_downloadermiddleware_redirect.py
View File

@ -1,7 +1,7 @@
import unittest
from scrapy.downloadermiddlewares.redirect import RedirectMiddleware, MetaRefreshMiddleware
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.exceptions import IgnoreRequest
from scrapy.http import Request, Response, HtmlResponse
from scrapy.utils.test import get_crawler

2
tests/test_downloadermiddleware_retry.py
View File

@ -7,7 +7,7 @@ from twisted.internet.error import TimeoutError, DNSLookupError, \
from scrapy import optional_features
from scrapy.downloadermiddlewares.retry import RetryMiddleware
from scrapy.xlib.tx import ResponseFailed
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request, Response
from scrapy.utils.test import get_crawler

2
tests/test_downloadermiddleware_stats.py
View File

@ -2,7 +2,7 @@ from unittest import TestCase
from scrapy.downloadermiddlewares.stats import DownloaderStats
from scrapy.http import Request, Response
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.utils.test import get_crawler

2
tests/test_downloadermiddleware_useragent.py
View File

@ -1,6 +1,6 @@
from unittest import TestCase
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
from scrapy.utils.test import get_crawler

2
tests/test_engine.py
View File

@ -22,7 +22,7 @@ from scrapy import signals
from scrapy.utils.test import get_crawler
from scrapy.xlib.pydispatch import dispatcher
from tests import tests_datadir
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.item import Item, Field
from scrapy.linkextractors import LinkExtractor
from scrapy.http import Request

2
tests/test_logformatter.py
View File

@ -1,6 +1,6 @@
import unittest
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.http import Request, Response
from scrapy.item import Item, Field
from scrapy.logformatter import LogFormatter

2
tests/test_pipeline_media.py
View File

@ -6,7 +6,7 @@ from twisted.internet import reactor
from twisted.internet.defer import Deferred, inlineCallbacks
from scrapy.http import Request, Response
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.utils.request import request_fingerprint
from scrapy.pipelines.media import MediaPipeline
from scrapy.utils.signal import disconnect_all

5
tests/test_spider.py
View File

@ -7,11 +7,10 @@ from testfixtures import LogCapture
from twisted.trial import unittest
from scrapy import signals
from scrapy.spider import Spider, BaseSpider
from scrapy.settings import Settings
from scrapy.http import Request, Response, TextResponse, XmlResponse, HtmlResponse
from scrapy.spiders.init import InitSpider
from scrapy.spiders import CrawlSpider, Rule, XMLFeedSpider, \
from scrapy.spiders import Spider, BaseSpider, CrawlSpider, Rule, XMLFeedSpider, \
CSVFeedSpider, SitemapSpider
from scrapy.linkextractors import LinkExtractor
from scrapy.exceptions import ScrapyDeprecationWarning
@ -116,7 +115,7 @@ class SpiderTest(unittest.TestCase):
def test_log(self):
spider = self.spider_class('example.com')
with mock.patch('scrapy.spider.Spider.logger') as mock_logger:
with mock.patch('scrapy.spiders.Spider.logger') as mock_logger:
spider.log('test log msg', 'INFO')
mock_logger.log.assert_called_once_with('INFO', 'test log msg')

2
tests/test_spiderloader/__init__.py
View File

@ -6,7 +6,7 @@ from zope.interface.verify import verifyObject
from twisted.trial import unittest
# ugly hack to avoid cyclic imports of scrapy.spider when running this test
# ugly hack to avoid cyclic imports of scrapy.spiders when running this test
# alone
from scrapy.interfaces import ISpiderLoader
from scrapy.spiderloader import SpiderLoader

2
tests/test_spiderloader/test_spiders/spider0.py
View File

@ -1,4 +1,4 @@
from scrapy.spider import Spider
from scrapy.spiders import Spider
class Spider0(Spider):
allowed_domains = ["scrapy1.org", "scrapy3.org"]

2
tests/test_spiderloader/test_spiders/spider1.py
View File

@ -1,4 +1,4 @@
from scrapy.spider import Spider
from scrapy.spiders import Spider
class Spider1(Spider):
name = "spider1"

2
tests/test_spiderloader/test_spiders/spider2.py
View File

@ -1,4 +1,4 @@
from scrapy.spider import Spider
from scrapy.spiders import Spider
class Spider2(Spider):
name = "spider2"

2
tests/test_spiderloader/test_spiders/spider3.py
View File

@ -1,4 +1,4 @@
from scrapy.spider import Spider
from scrapy.spiders import Spider
class Spider3(Spider):
name = "spider3"

2
tests/test_spidermiddleware_depth.py
View File

@ -2,7 +2,7 @@ from unittest import TestCase
from scrapy.spidermiddlewares.depth import DepthMiddleware
from scrapy.http import Response, Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.statscollectors import StatsCollector
from scrapy.utils.test import get_crawler

2
tests/test_spidermiddleware_httperror.py
View File

@ -7,7 +7,7 @@ from twisted.internet import defer
from scrapy.utils.test import get_crawler
from tests.mockserver import MockServer
from scrapy.http import Response, Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.spidermiddlewares.httperror import HttpErrorMiddleware, HttpError
from scrapy.settings import Settings

2
tests/test_spidermiddleware_offsite.py
View File

@ -3,7 +3,7 @@ from unittest import TestCase
from six.moves.urllib.parse import urlparse
from scrapy.http import Response, Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.spidermiddlewares.offsite import OffsiteMiddleware
from scrapy.utils.test import get_crawler

2
tests/test_spidermiddleware_referer.py
View File

@ -1,7 +1,7 @@
from unittest import TestCase
from scrapy.http import Response, Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.spidermiddlewares.referer import RefererMiddleware

2
tests/test_spidermiddleware_urllength.py
View File

@ -2,7 +2,7 @@ from unittest import TestCase
from scrapy.spidermiddlewares.urllength import UrlLengthMiddleware
from scrapy.http import Response, Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
class TestUrlLengthMiddleware(TestCase):

2
tests/test_spiderstate.py
View File

@ -3,7 +3,7 @@ from datetime import datetime
from twisted.trial import unittest
from scrapy.extensions.spiderstate import SpiderState
from scrapy.spider import Spider
from scrapy.spiders import Spider
class SpiderStateTest(unittest.TestCase):

2
tests/test_stats.py
View File

@ -1,6 +1,6 @@
import unittest
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.statscollectors import StatsCollector, DummyStatsCollector
from scrapy.utils.test import get_crawler

2
tests/test_toplevel.py
View File

@ -21,7 +21,7 @@ class ToplevelTestCase(TestCase):
self.assertIs(scrapy.FormRequest, FormRequest)
def test_spider_shortcut(self):
from scrapy.spider import Spider
from scrapy.spiders import Spider
self.assertIs(scrapy.Spider, Spider)
def test_selector_shortcut(self):

2
tests/test_utils_reqser.py
View File

@ -1,7 +1,7 @@
import unittest
from scrapy.http import Request
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.utils.reqser import request_to_dict, request_from_dict
class RequestSerializationTest(unittest.TestCase):

2
tests/test_utils_url.py
View File

@ -1,6 +1,6 @@
import unittest
from scrapy.spider import Spider
from scrapy.spiders import Spider
from scrapy.utils.url import url_is_from_any_domain, url_is_from_spider, canonicalize_url
__doctests__ = ['scrapy.utils.url']