4.4 KiB
Frequently Asked Questions
How does Scrapy compare to BeautifulSoul or lxml?
BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an application framework for writing web spiders that crawl web sites and extract data from them.
Scrapy provides a built-in mechanism for extracting data (called :ref:`selectors <topics-selectors>`) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they're just parsing libraries which can be imported and used from any Python code.
System Message: ERROR/3 (<stdin>, line 13); backlink
Unknown interpreted text role "ref".In other words, comparing BeautifulSoup or lxml to Scrapy is like comparing urllib or urlparse to Django (a popular Python web application framework).
Does Scrapy work with Python 3.0?
No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment Scrapy works with Python 2.5 or 2.6.
Did Scrapy "steal" X from Django?
Probably, but we don't like that word. We think Django is a great open source project and an example to follow, so we've used it as an inspiration for Scrapy.
We believe that, if something is already done well, there's no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve.
We'd be proud if Scrapy serves as an inspiration for other projects. Feel free to steal from us!
Does Scrapy work with HTTP proxies?
No. Support for HTTP proxies is not currently implemented in Scrapy, but it will be in the future. For more information about this, follow this ticket. Setting the http_proxy environment variable won't work because Twisted (the library used by Scrapy to download pages) doesn't support it. See this Twisted ticket for more info.
Scrapy crashes with: ImportError: No module named win32api
You need to install pywin32 because of this Twisted bug.
How can I simulate a user login in my spider?
See :ref:`topics-request-response-ref-request-userlogin`.
System Message: ERROR/3 (<stdin>, line 75); backlink
Unknown interpreted text role "ref".Can I crawl in depth-first order instead of breadth-first order?
Yes, there's a setting for that: :setting:`SCHEDULER_ORDER`.
System Message: ERROR/3 (<stdin>, line 80); backlink
Unknown interpreted text role "setting".My Scrapy crawler has memory leaks. What can I do?
See :ref:`topics-leaks`.
System Message: ERROR/3 (<stdin>, line 85); backlink
Unknown interpreted text role "ref".Can I use Basic HTTP Authentication in my spiders?
Yes, see :class:`~scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware`.
System Message: ERROR/3 (<stdin>, line 90); backlink
Unknown interpreted text role "class".Why does Scrapy download pages in English instead of my native language?
Try changing the default Accept-Language request header by overriding the :setting:`DEFAULT_REQUEST_HEADERS` setting.
System Message: ERROR/3 (<stdin>, line 95); backlink
Unknown interpreted text role "setting".Where can I find some example code using Scrapy?
Scrapy comes with a built-in, fully functional project to scrape the Google Directory. You can find it in the examples/googledir directory of the Scrapy distribution.
Also, there is a public repository of spiders called Community Spiders.
Finally, you can find some example code for performing not-so-trivial tasks in the Scrapy Recipes page.