8.4 KiB
Frequently Asked Questions
How does Scrapy compare to BeautifulSoul or lxml?
BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an application framework for writing web spiders that crawl web sites and extract data from them.
Scrapy provides a built-in mechanism for extracting data (called :ref:`selectors <topics-selectors>`) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they're just parsing libraries which can be imported and used from any Python code.
System Message: ERROR/3 (<stdin>, line 13); backlink
Unknown interpreted text role "ref".In other words, comparing BeautifulSoup (or lxml) to Scrapy is like comparing jinja2 to Django.
What Python versions does Scrapy support?
Scrapy runs in Python 2.5, 2.6 and 2.6. But it's recommended you use Python 2.6 or above, since the Python 2.5 standard library has a few bugs in their URL handling libraries. Some of these Python 2.5 bugs not only affect Scrapy but any user code, such as spiders. You can see a list of Python 2.5 bugs that affect Scrapy in the issue tracker.
Does Scrapy work with Python 3.0?
No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment, Scrapy works with Python 2.5, 2.6 and 2.7.
System Message: ERROR/3 (<stdin>, line 46)
Unknown directive type "seealso".
.. seealso:: :ref:`faq-python-versions`.
Did Scrapy "steal" X from Django?
Probably, but we don't like that word. We think Django is a great open source project and an example to follow, so we've used it as an inspiration for Scrapy.
We believe that, if something is already done well, there's no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve.
We'd be proud if Scrapy serves as an inspiration for other projects. Feel free to steal from us!
Does Scrapy work with HTTP proxies?
Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP Proxy downloader middleware. See :class:`~scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware`.
System Message: ERROR/3 (<stdin>, line 70); backlink
Unknown interpreted text role "class".Scrapy crashes with: ImportError: No module named win32api
You need to install pywin32 because of this Twisted bug.
How can I simulate a user login in my spider?
See :ref:`topics-request-response-ref-request-userlogin`.
System Message: ERROR/3 (<stdin>, line 85); backlink
Unknown interpreted text role "ref".Can I crawl in breadth-first order instead of depth-first order?
Yes, there's a setting for that: :setting:`SCHEDULER_ORDER`.
System Message: ERROR/3 (<stdin>, line 90); backlink
Unknown interpreted text role "setting".My Scrapy crawler has memory leaks. What can I do?
See :ref:`topics-leaks`.
System Message: ERROR/3 (<stdin>, line 95); backlink
Unknown interpreted text role "ref".Also, Python has a builtin memory leak issue which is described in :ref:`topics-leaks-without-leaks`.
System Message: ERROR/3 (<stdin>, line 97); backlink
Unknown interpreted text role "ref".How can I make Scrapy consume less memory?
See previous question.
Can I use Basic HTTP Authentication in my spiders?
Yes, see :class:`~scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware`.
System Message: ERROR/3 (<stdin>, line 108); backlink
Unknown interpreted text role "class".Why does Scrapy download pages in English instead of my native language?
Try changing the default Accept-Language request header by overriding the :setting:`DEFAULT_REQUEST_HEADERS` setting.
System Message: ERROR/3 (<stdin>, line 113); backlink
Unknown interpreted text role "setting".Where can I find some example code using Scrapy?
Scrapy comes with a built-in, fully functional project to scrape the Google Directory. You can find it in the examples/googledir directory of the Scrapy distribution.
Also, there's a site for sharing code snippets (spiders, middlewares, extensions) called Scrapy snippets.
Finally, you can find some example code for performing not-so-trivial tasks in the Scrapy Recipes wiki page.
Can I run a spider without creating a project?
Yes. You can use the :command:`runspider` command. For example, if you have a spider written in a my_spider.py file you can run it with:
System Message: ERROR/3 (<stdin>, line 140); backlink
Unknown interpreted text role "command".scrapy runspider my_spider.py
See :command:`runspider` command for more info.
System Message: ERROR/3 (<stdin>, line 145); backlink
Unknown interpreted text role "command".I get "Filtered offsite request" messages. How can I fix them?
Those messages (logged with DEBUG level) don't necessarily mean there is a problem, so you may not need to fix them.
Those message are thrown by the Offsite Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider.
For more info see: :class:`~scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware`.
System Message: ERROR/3 (<stdin>, line 157); backlink
Unknown interpreted text role "class".What is the recommended way to deploy a Scrapy crawler in production?
System Message: ERROR/3 (<stdin>, line 163); backlink
Unknown interpreted text role "ref".Can I use JSON for large exports?
It'll depend on how large your output is. See :ref:`this warning <json-with-large-data>` in :class:`~scrapy.contrib.exporter.JsonItemExporter` documentation.
System Message: ERROR/3 (<stdin>, line 168); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 168); backlink
Unknown interpreted text role "class".Can I return (Twisted) deferreds from signal handlers?
Some signals support returning deferreds from their handlers, others don't. See the :ref:`topics-signals-ref` to know which ones.
System Message: ERROR/3 (<stdin>, line 175); backlink
Unknown interpreted text role "ref".What does the response status code 999 means?
999 is a custom reponse status code used by Yahoo sites to throttle requests. Try slowing down the crawling speed by using a download delay of 2 (or higher) in your spider:
class MySpider(CrawlSpider):
name = 'myspider'
DOWNLOAD_DELAY = 2
# [ ... rest of the spider code ... ]
Or by setting a global download delay in your project with the :setting:`DOWNLOAD_DELAY` setting.
System Message: ERROR/3 (<stdin>, line 193); backlink
Unknown interpreted text role "setting".Can I call pdb.set_trace() from my spiders to debug them?
Yes, but you can also use the Scrapy shell which allows you too quickly analyze (and even modify) the response being processed by your spider, which is, quite often, more useful than plain old pdb.set_trace().
For more info see :ref:`topics-shell-inspect-response`.
System Message: ERROR/3 (<stdin>, line 203); backlink
Unknown interpreted text role "ref".Simplest way to dump all my scraped items into a JSON/CSV/XML file?
To dump into a JSON file:
scrapy crawl myspider --set FEED_URI=items.json --set FEED_FORMAT=json
To dump into a CSV file:
scrapy crawl myspider --set FEED_URI=items.csv --set FEED_FORMAT=csv
To dump into a XML file:
scrapy crawl myspider --set FEED_URI=items.xml --set FEED_FORMAT=xml
For more information see :ref:`topics-feed-exports`
System Message: ERROR/3 (<stdin>, line 220); backlink
Unknown interpreted text role "ref".What's this huge cryptic __VIEWSTATE parameter used in some forms?
The __VIEWSTATE parameter is used in sites built with ASP.NET/VB.NET. For more info on how it works see this page. Also, here's an example spider which scrapes one of these sites.