diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index ecde3da43..b4d058754 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -138,7 +138,7 @@ To disable cookies use: Disable retries =============== -Retrying failed HTTP requests can slow down the crawls substantially, specially +Retrying failed HTTP requests can slow down the crawls substantially, especially when sites causes are very slow (or fail) to respond, thus causing a timeout error which gets retried many times, unnecessarily, preventing crawler capacity to be reused for other domains. diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst index d2ddde207..0fbd22a27 100644 --- a/docs/topics/extensions.rst +++ b/docs/topics/extensions.rst @@ -267,7 +267,7 @@ CLOSESPIDER_TIMEOUT Default: ``0`` An integer which specifies a number of seconds. If the spider remains open for -more than that number of second, it will be automatically closed with the +more than that number of seconds, it will be automatically closed with the reason ``closespider_timeout``. If zero (or non set), spiders won't be closed by timeout. diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 56177ba4e..79d5bcce9 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -281,7 +281,7 @@ finishes before starting the next one: Distributed crawls ================== -Scrapy doesn't provide any built-in facility for running crawls in a distribute +Scrapy doesn't provide any built-in facility for running crawls in a distributed (multi-server) manner. However, there are some ways to distribute crawls, which vary depending on how you plan to distribute them. @@ -289,10 +289,10 @@ If you have many spiders, the obvious way to distribute the load is to setup many Scrapyd instances and distribute spider runs among those. If you instead want to run a single (big) spider through many machines, what -you usually do is partition the urls to crawl and send them to each separate +you usually do is partition the URLs to crawl and send them to each separate spider. Here is a concrete example: -First, you prepare the list of urls to crawl and put them into separate +First, you prepare the list of URLs to crawl and put them into separate files/urls:: http://somedomain.com/urls-to-crawl/spider1/part1.list @@ -319,7 +319,7 @@ consider contacting `commercial support`_ if in doubt. Here are some tips to keep in mind when dealing with these kinds of sites: -* rotate your user agent from a pool of well-known ones from browsers (google +* rotate your user agent from a pool of well-known ones from browsers (Google around to get a list of them) * disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use cookies to spot bot behaviour diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst index 85a08cebd..8ae8ff512 100644 --- a/docs/topics/shell.rst +++ b/docs/topics/shell.rst @@ -24,7 +24,7 @@ If you have `IPython`_ installed, the Scrapy shell will use it (instead of the standard Python console). The `IPython`_ console is much more powerful and provides smart auto-completion and colorized output, among other things. -We highly recommend you install `IPython`_, specially if you're working on +We highly recommend you install `IPython`_, especially if you're working on Unix systems (where `IPython`_ excels). See the `IPython installation guide`_ for more info. diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 8240d5d4b..390a5f541 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -628,7 +628,7 @@ XMLFeedSpider This method is called for the nodes matching the provided tag name (``itertag``). Receives the response and an :class:`~scrapy.Selector` for each node. Overriding this - method is mandatory. Otherwise, you spider won't work. This method + method is mandatory. Otherwise, your spider won't work. This method must return an :ref:`item object `, a :class:`~scrapy.Request` object, or an iterable containing any of them.