mirror of https://github.com/scrapy/scrapy.git
42dc34f924
Install twisted trunk because released twisted doesn't install properly, see https://twistedmatrix.com/trac/ticket/6539. Also, test-requirements are removed for 3.3 because mitmproxy doesn't install in Python 3.3, and mock is in Python 3.3 standard library. Note that trial scrapy is not doing what is expected because twisted doesn't have trial runner for 3.3 yet, so this command will most likely call "trial" outside tox virtualenv and will use incorrect interpreter. |
||
|---|---|---|
| artwork | ||
| bin | ||
| debian | ||
| docs | ||
| extras | ||
| scrapy | ||
| sep | ||
| .coveragerc | ||
| .gitignore | ||
| .travis.yml | ||
| AUTHORS | ||
| CONTRIBUTING.md | ||
| INSTALL | ||
| LICENSE | ||
| MANIFEST.in | ||
| Makefile.buildbot | ||
| NEWS | ||
| README.rst | ||
| requirements.txt | ||
| setup.cfg | ||
| setup.py | ||
| tests-requirements.txt | ||
| tox.ini | ||
README.rst
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
</head>
</html>
Scrapy
Overview
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
For more information including a list of features check the Scrapy homepage at: http://scrapy.org
Requirements
- Python 2.7
- Works on Linux, Windows, Mac OSX, BSD
Install
The quick way:
pip install scrapy
For more details see the install section in the documentation: http://doc.scrapy.org/en/latest/intro/install.html
Releases
You can download the latest stable and development releases from: http://scrapy.org/download/
Documentation
Documentation is available online at http://doc.scrapy.org/ and in the docs directory.