19 KiB
Common Practices
This section documents common practices when using Scrapy. These are things that cover many topics and don't often fall into any other specific section.
Run Scrapy from a script
You can use the :ref:`API <topics-api>` to run Scrapy from a script, instead of the typical way of running Scrapy via scrapy crawl.
System Message: ERROR/3 (<stdin>, line 17); backlink
Unknown interpreted text role "ref".Remember that Scrapy requires a Twisted reactor or (with :setting:`TWISTED_REACTOR_ENABLED` set to False) an asyncio event loop, so you need to run one of those in your script for it to work (helpers described below can do it for you).
System Message: ERROR/3 (<stdin>, line 20); backlink
Unknown interpreted text role "setting".The first utility you can use to run your spiders is :class:`scrapy.crawler.AsyncCrawlerProcess` or :class:`scrapy.crawler.CrawlerProcess`. These classes will start a Twisted reactor for you, configuring the logging and setting shutdown handlers. These classes are the ones used by all Scrapy commands. They have similar functionality, differing in their asynchronous API style: :class:`~scrapy.crawler.AsyncCrawlerProcess` returns coroutines from its asynchronous methods while :class:`~scrapy.crawler.CrawlerProcess` returns :class:`~twisted.internet.defer.Deferred` objects.
System Message: ERROR/3 (<stdin>, line 25); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 25); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 25); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 25); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 25); backlink
Unknown interpreted text role "class".Here's an example showing how to run a single spider with it.
System Message: WARNING/2 (<stdin>, line 37)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from scrapy.crawler import AsyncCrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process = AsyncCrawlerProcess(
settings={
"FEEDS": {
"items.json": {"format": "json"},
},
}
)
process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
You can define :ref:`settings <topics-settings>` within the dictionary passed to :class:`~scrapy.crawler.AsyncCrawlerProcess`. Make sure to check the :class:`~scrapy.crawler.AsyncCrawlerProcess` documentation to get acquainted with its usage details.
System Message: ERROR/3 (<stdin>, line 59); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 59); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 59); backlink
Unknown interpreted text role "class".If you are inside a Scrapy project there are some additional helpers you can use to import those components within the project. You can automatically import your spiders passing their name to :class:`~scrapy.crawler.AsyncCrawlerProcess`, and use :func:`scrapy.utils.project.get_project_settings` to get a :class:`~scrapy.settings.Settings` instance with your project settings.
System Message: ERROR/3 (<stdin>, line 64); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 64); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 64); backlink
Unknown interpreted text role "class".What follows is a working example of how to do that, using the testspiders project as example.
System Message: WARNING/2 (<stdin>, line 74)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from scrapy.crawler import AsyncCrawlerProcess
from scrapy.utils.project import get_project_settings
process = AsyncCrawlerProcess(get_project_settings())
# 'followall' is the name of one of the spiders of the project.
process.crawl("followall", domain="scrapy.org")
process.start() # the script will block here until the crawling is finished
There's another Scrapy utility that provides more control over the crawling process: :class:`scrapy.crawler.AsyncCrawlerRunner` or :class:`scrapy.crawler.CrawlerRunner`. These classes are thin wrappers that encapsulate some simple helpers to run multiple crawlers, but they won't start or interfere with existing reactors in any way. Just like :class:`scrapy.crawler.AsyncCrawlerProcess` and :class:`scrapy.crawler.CrawlerProcess` they differ in their asynchronous API style.
System Message: ERROR/3 (<stdin>, line 85); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 85); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 85); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 85); backlink
Unknown interpreted text role "class".When using these classes the reactor should be explicitly run after scheduling your spiders. It's recommended that you use :class:`~scrapy.crawler.AsyncCrawlerRunner` or :class:`~scrapy.crawler.CrawlerRunner` instead of :class:`~scrapy.crawler.AsyncCrawlerProcess` or :class:`~scrapy.crawler.CrawlerProcess` if your application is already using Twisted and you want to run Scrapy in the same reactor.
System Message: ERROR/3 (<stdin>, line 94); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 94); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 94); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 94); backlink
Unknown interpreted text role "class".If you want to stop the reactor or run any other code right after the spider finishes you can do that after the task returned from :meth:`AsyncCrawlerRunner.crawl() <scrapy.crawler.AsyncCrawlerRunner.crawl>` completes (or the Deferred returned from :meth:`CrawlerRunner.crawl() <scrapy.crawler.CrawlerRunner.crawl>` fires). In the simplest case you can also use :func:`twisted.internet.task.react` to start and stop the reactor, though it may be easier to just use :class:`~scrapy.crawler.AsyncCrawlerProcess` or :class:`~scrapy.crawler.CrawlerProcess` instead.
System Message: ERROR/3 (<stdin>, line 102); backlink
Unknown interpreted text role "meth".System Message: ERROR/3 (<stdin>, line 102); backlink
Unknown interpreted text role "meth".System Message: ERROR/3 (<stdin>, line 102); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 102); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 102); backlink
Unknown interpreted text role "class".Here's an example of using :class:`~scrapy.crawler.AsyncCrawlerRunner` together with simple reactor management code:
System Message: ERROR/3 (<stdin>, line 111); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 114)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.defer import deferred_f_from_coro_f
from scrapy.utils.log import configure_logging
from scrapy.utils.reactor import install_reactor
from twisted.internet.task import react
class MySpider(scrapy.Spider):
# Your spider definition
...
async def crawl(_):
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = AsyncCrawlerRunner()
await runner.crawl(MySpider) # completes when the spider finishes
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
react(deferred_f_from_coro_f(crawl))
Same example but using :class:`~scrapy.crawler.CrawlerRunner` and a different reactor (:class:`~scrapy.crawler.AsyncCrawlerRunner` only works with :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`):
System Message: ERROR/3 (<stdin>, line 138); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 138); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 138); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 142)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.reactor import install_reactor
from twisted.internet.task import react
class MySpider(scrapy.Spider):
custom_settings = {
"TWISTED_REACTOR": "twisted.internet.epollreactor.EPollReactor",
}
# Your spider definition
...
def crawl(_):
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = CrawlerRunner()
d = runner.crawl(MySpider)
return d # this Deferred fires when the spider finishes
install_reactor("twisted.internet.epollreactor.EPollReactor")
react(crawl)
System Message: ERROR/3 (<stdin>, line 169)
Unknown directive type "seealso".
.. seealso:: :doc:`twisted:core/howto/reactor-basics`
And here are examples of using these classes with :setting:`TWISTED_REACTOR_ENABLED` set to False.
System Message: ERROR/3 (<stdin>, line 171); backlink
Unknown interpreted text role "setting".Simple usage of :class:`~scrapy.crawler.AsyncCrawlerProcess`:
System Message: ERROR/3 (<stdin>, line 174); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 176)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from scrapy.crawler import AsyncCrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process = AsyncCrawlerProcess(
settings={
"TWISTED_REACTOR_ENABLED": False,
}
)
process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
With TWISTED_REACTOR_ENABLED=False you can use several instances of :class:`~scrapy.crawler.AsyncCrawlerProcess` in the same process:
System Message: ERROR/3 (<stdin>, line 196); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 199)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from scrapy.crawler import AsyncCrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process1 = AsyncCrawlerProcess(
settings={
"TWISTED_REACTOR_ENABLED": False,
}
)
process1.crawl(MySpider)
process1.start()
process2 = AsyncCrawlerProcess(
settings={
"TWISTED_REACTOR_ENABLED": False,
}
)
process2.crawl(MySpider)
process2.start()
Using :func:`asyncio.run` with :class:`~scrapy.crawler.AsyncCrawlerRunner`:
System Message: ERROR/3 (<stdin>, line 226); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 226); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 228)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import asyncio
import scrapy
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.log import configure_logging
class MySpider(scrapy.Spider):
# Your spider definition
...
async def main():
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = AsyncCrawlerRunner(settings={"TWISTED_REACTOR_ENABLED": False})
await runner.crawl(MySpider) # completes when the spider finishes
asyncio.run(main())
Running spiders inside existing applications
You may want to run Scrapy spiders inside an existing application. In simple cases (e.g. task queues that spawn a process for every task, or applications that can execute tasks synchronously in the same process) you can use the same approach as for standalone scripts (see :ref:`run-from-script`). More complex cases, e.g. asynchronous web applications, have additional caveats and limitations.
System Message: ERROR/3 (<stdin>, line 255); backlink
Unknown interpreted text role "ref".If the application runs its own Twisted reactor, you can use :class:`~scrapy.crawler.AsyncCrawlerRunner` or :class:`~scrapy.crawler.CrawlerRunner` to run spiders using this reactor, see :ref:`run-from-script` for examples.
System Message: ERROR/3 (<stdin>, line 262); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 262); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 262); backlink
Unknown interpreted text role "ref".If the application doesn't run a Twisted reactor or an asyncio event loop (for example, a Django web app deployed with a WSGI server such as uWSGI), you can use :class:`~scrapy.crawler.AsyncCrawlerProcess` with :setting:`TWISTED_REACTOR_ENABLED` set to False, so that Scrapy starts and stops an asyncio event loop for every spider run:
System Message: ERROR/3 (<stdin>, line 267); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 267); backlink
Unknown interpreted text role "setting".System Message: WARNING/2 (<stdin>, line 273)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from django.http import HttpResponse
from scrapy.crawler import AsyncCrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
def crawl_view(request):
process = AsyncCrawlerProcess(settings={"TWISTED_REACTOR_ENABLED": False})
process.crawl(MySpider)
process.start() # returns when the spider finishes
return HttpResponse("Crawling finished")
If the application runs its own asyncio event loop (for example, a Django web app deployed with an ASGI server such as uvicorn), you can use :class:`~scrapy.crawler.AsyncCrawlerRunner` with :setting:`TWISTED_REACTOR_ENABLED` set to False, so that Scrapy uses the existing event loop:
System Message: ERROR/3 (<stdin>, line 291); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 291); backlink
Unknown interpreted text role "setting".System Message: WARNING/2 (<stdin>, line 297)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from django.http import HttpResponse
from scrapy.crawler import AsyncCrawlerRunner
class MySpider(scrapy.Spider):
# Your spider definition
...
async def crawl_view(request):
runner = AsyncCrawlerRunner(settings={"TWISTED_REACTOR_ENABLED": False})
await runner.crawl(MySpider) # completes when the spider finishes
return HttpResponse("Crawling finished")
Note
Running Scrapy without a Twisted reactor is experimental and has some limitations, described in :ref:`asyncio-without-reactor`.
System Message: ERROR/3 (<stdin>, line 314); backlink
Unknown interpreted text role "ref".Running spiders in Jupyter notebooks
You can run Scrapy spiders in Jupyter notebooks. You need to use :class:`~scrapy.crawler.AsyncCrawlerRunner` with :setting:`TWISTED_REACTOR_ENABLED` set to False for this, so that Scrapy uses the event loop provided by the notebook kernel. As :class:`~scrapy.crawler.AsyncCrawlerRunner` doesn't configure logging, and you most likely want to see the spider log in the notebook, you should call :func:`scrapy.utils.log.configure_logging`. Here is a full example, which supports rerunning both as a single cell and as separate cells:
System Message: ERROR/3 (<stdin>, line 322); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 322); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 322); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 322); backlink
Unknown interpreted text role "func".System Message: WARNING/2 (<stdin>, line 331)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from scrapy import Spider
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.log import configure_logging
configure_logging()
class BooksSpider(Spider):
name = "books"
start_urls = ["https://books.toscrape.com"]
def parse(self, response):
for book in response.css("h3"):
yield {"title": book.css("a::attr(title)").get()}
runner = AsyncCrawlerRunner({"TWISTED_REACTOR_ENABLED": False})
await runner.crawl(BooksSpider)
Note
Running Scrapy without a Twisted reactor is experimental and has some limitations, described in :ref:`asyncio-without-reactor`.
System Message: ERROR/3 (<stdin>, line 352); backlink
Unknown interpreted text role "ref".Running multiple spiders in the same process
By default, Scrapy runs a single spider per process when you run scrapy crawl. However, Scrapy supports running multiple spiders per process using the :ref:`internal API <topics-api>`.
System Message: ERROR/3 (<stdin>, line 360); backlink
Unknown interpreted text role "ref".Here is an example that runs multiple spiders simultaneously:
System Message: WARNING/2 (<stdin>, line 366)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from scrapy.crawler import AsyncCrawlerProcess
from scrapy.utils.project import get_project_settings
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
settings = get_project_settings()
process = AsyncCrawlerProcess(settings)
process.crawl(MySpider1)
process.crawl(MySpider2)
process.start() # the script will block here until all crawling jobs are finished
Same example using :class:`~scrapy.crawler.AsyncCrawlerRunner`:
System Message: ERROR/3 (<stdin>, line 389); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 391)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.defer import deferred_f_from_coro_f
from scrapy.utils.log import configure_logging
from scrapy.utils.reactor import install_reactor
from twisted.internet.task import react
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
async def crawl(_):
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = AsyncCrawlerRunner()
runner.crawl(MySpider1)
runner.crawl(MySpider2)
await runner.join() # completes when both spiders finish
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
react(deferred_f_from_coro_f(crawl))
Same example but running the spiders sequentially by awaiting until each one finishes before starting the next one:
System Message: WARNING/2 (<stdin>, line 426)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.defer import deferred_f_from_coro_f
from scrapy.utils.log import configure_logging
from scrapy.utils.reactor import install_reactor
from twisted.internet.task import react
class MySpider1(scrapy.Spider):
# Your first spider definition
...
class MySpider2(scrapy.Spider):
# Your second spider definition
...
async def crawl(_):
configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})
runner = AsyncCrawlerRunner()
await runner.crawl(MySpider1)
await runner.crawl(MySpider2)
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
react(deferred_f_from_coro_f(crawl))
Note
When running multiple spiders in the same process, :ref:`logging settings <logging-settings>` and :ref:`reactor settings <reactor-settings>` should not have a different value per spider, and :ref:`pre-crawler settings <pre-crawler-settings>` cannot be defined per spider.
System Message: ERROR/3 (<stdin>, line 456); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 456); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 456); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 461)
Unknown directive type "seealso".
.. seealso:: :ref:`run-from-script`.
Distributed crawls
Scrapy doesn't provide any built-in facility for running crawls in a distributed (multi-server) manner. However, there are some ways to distribute crawls, which vary depending on how you plan to distribute them.
If you have many spiders, the obvious way to distribute the load is to setup many Scrapyd instances and distribute spider runs among those.
If you instead want to run a single (big) spider through many machines, what you usually do is partition the URLs to crawl and send them to each separate spider. Here is a concrete example:
First, you prepare the list of URLs to crawl and put them into separate files/urls:
http://somedomain.com/urls-to-crawl/spider1/part1.list http://somedomain.com/urls-to-crawl/spider1/part2.list http://somedomain.com/urls-to-crawl/spider1/part3.list
Then you fire a spider run on 3 different Scrapyd servers. The spider would receive a (spider) argument part with the number of the partition to crawl:
curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1 curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2 curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
Reducing startup time in large projects
When running a spider with scrapy crawl, Scrapy loads all modules listed in :setting:`SPIDER_MODULES` to find the target spider. In large projects with many spiders, this can noticeably increase startup time and memory usage.
System Message: ERROR/3 (<stdin>, line 501); backlink
Unknown interpreted text role "setting".To avoid loading every spider module, override :setting:`SPIDER_MODULES` on the command line to point only to the module that contains the spider you want to run:
System Message: ERROR/3 (<stdin>, line 505); backlink
Unknown interpreted text role "setting".System Message: WARNING/2 (<stdin>, line 509)
Cannot analyze code. Pygments package not found.
.. code-block:: shell
scrapy crawl myspider -s SPIDER_MODULES=myproject.spiders.myspider
Because :setting:`SPIDER_MODULES` is a list setting, you can include multiple modules by separating them with commas.
System Message: ERROR/3 (<stdin>, line 513); backlink
Unknown interpreted text role "setting".Avoiding getting banned
Some websites implement certain measures to prevent bots from crawling them, with varying degrees of sophistication. Getting around those measures can be difficult and tricky, and may sometimes require special infrastructure. Please consider contacting commercial support if in doubt.
Here are some tips to keep in mind when dealing with these kinds of sites:
rotate your user agent from a pool of well-known ones from browsers (Google around to get a list of them)
disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use cookies to spot bot behaviour
System Message: ERROR/3 (<stdin>, line 530); backlink
Unknown interpreted text role "setting".
use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
System Message: ERROR/3 (<stdin>, line 532); backlink
Unknown interpreted text role "setting".
if possible, use Common Crawl to fetch pages, instead of hitting the sites directly
use a pool of rotating IPs. For example, the free Tor project or paid services like ProxyMesh. An open source alternative is scrapoxy, a super proxy that you can attach your own proxies to.
for HTTPS websites, if blocking appears related to TLS behavior, consider adjusting the :setting:`DOWNLOAD_TLS_MIN_VERSION` and :setting:`DOWNLOAD_TLS_MAX_VERSION` settings, since some websites may respond differently depending on the TLS method used by the client.
System Message: ERROR/3 (<stdin>, line 538); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 538); backlink
Unknown interpreted text role "setting".
use a ban avoidance service, such as Zyte API, which provides a Scrapy plugin and additional features, like AI web scraping
If you are still unable to prevent your bot getting banned, consider contacting commercial support.
Static analysis
Consider using :doc:`scrapy-lint <scrapy-lint:index>`, a linter for Scrapy projects that detects common mistakes and anti-patterns.
System Message: ERROR/3 (<stdin>, line 554); backlink
Unknown interpreted text role "doc".