RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Simplified installation guide, including lxml as alternative dependency to libxml2. Closes #280

This commit is contained in:
Pablo Hoffman 2010-11-17 21:32:23 -02:00
parent 2897061b98
commit ac007802d6
2 changed files with 116 additions and 177 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/faq.rst
View File

@ -41,8 +41,9 @@ Does Scrapy work with Python 3.0?
---------------------------------
No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment,
Scrapy works with Python 2.5, 2.6 and 2.7. See also:
:ref:`faq-python-versions`.
Scrapy works with Python 2.5, 2.6 and 2.7.
.. seealso:: :ref:`faq-python-versions`.
Did Scrapy "steal" X from Django?
---------------------------------

288
docs/intro/install.rst
View File

@ -4,16 +4,7 @@
Installation guide
==================
This document describes how to install Scrapy on Linux, Windows and Mac OS X
systems and it consists of the following 3 steps:
* :ref:`intro-install-step1`
* :ref:`intro-install-step2`
* :ref:`intro-install-step3`
If you use Ubuntu (9.10 or above), you can use the :ref:`Ubuntu packages
<topics-ubuntu>` to install Scrapy more easily, skipping these steps. The
Ubuntu packages are also kept updated with latest bug fixes.
This document describes how to install Scrapy on Linux, Windows and Mac OS X.
.. _intro-install-requirements:
@ -25,138 +16,51 @@ Requirements
* `Twisted`_ 2.5.0, 8.0 or above (Windows users: you'll need to install
`Zope.Interface`_ and maybe `pywin32`_ because of `this Twisted bug`_)
* `libxml2`_ (versions prior to 2.6.28 are known to have problems parsing certain malformed HTML, and have also been reported to contain leaks, so 2.6.28 or above is highly recommended)
* `lxml`_ or `libxml2`_ (if using `libxml2`_, version 2.6.28 or above is highly recommended)
* `simplejson`_ (not required if using Python 2.6 or above)
.. _Python: http://www.python.org
.. _Twisted: http://twistedmatrix.com
.. _libxml2: http://xmlsoft.org
.. _pywin32: http://sourceforge.net/projects/pywin32/
.. _simplejson: http://pypi.python.org/pypi/simplejson/
.. _Zope.Interface: http://pypi.python.org/pypi/zope.interface#download
.. _this Twisted bug: http://twistedmatrix.com/trac/ticket/3707
* `pyopenssl <http://pyopenssl.sourceforge.net>`_ (for HTTPS support. Optional,
but highly recommended)
Optional:
.. _intro-install-python:
* `pyopenssl <http://pyopenssl.sourceforge.net>`_ (for HTTPS support, highly recommended)
Install Python
==============
.. _intro-install-step1:
First, you need to install Python, if you haven't done so already.
Step 1. Install Python
======================
Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at
http://www.python.org/download/
Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at http://www.python.org/download/
.. seealso:: :ref:`faq-python-versions`
.. highlight:: sh
.. _intro-install-step2:
.. _intro-install-scrapy:
Step 2. Install required libraries
==================================
Install Scrapy
==============
The procedure for installing the required third party libraries depends on the
platform and operating system you use.
There are many ways to install Scrapy. Pick the one you feel more comfortable
with.
Ubuntu/Debian
-------------
If you're running Ubuntu/Debian Linux, run the following command as root::
apt-get install python-twisted python-libxml2
To install optional libraries::
apt-get install python-pyopenssl python-simplejson
Arch Linux
----------
If you are running Arch Linux, run the following command as root::
pacman -S twisted libxml2
To install optional libraries::
pacman -S pyopenssl python-simplejson
Mac OS X
--------
First, download `Twisted for Mac`_.
.. _Twisted for Mac: http://twistedmatrix.com/trac/wiki/Downloads#MacOSX
Mac OS X ships an ``libxml2`` version too old to be used by Scrapy. Also, by
looking on the web it seems that installing ``libxml2`` on MacOSX is a bit of a
challenge. Here is a way to achieve this, though not acceptable on the long
run:
1. Fetch the following libxml2 and libxslt packages:
ftp://xmlsoft.org/libxml2/libxml2-2.7.3.tar.gz
ftp://xmlsoft.org/libxml2/libxslt-1.1.24.tar.gz
2. Extract, build and install them both with::
./configure --with-python=/Library/Frameworks/Python.framework/Versions/2.5/
make
sudo make install
Replacing ``/Library/Frameworks/Python.framework/Version/2.5/`` with your
current python framework location.
3. Install libxml2 Python bidings with::
cd libxml2-2.7.3/python
sudo make install
The libraries and modules should be installed in something like
/usr/local/lib/python2.5/site-packages. Add it to your ``PYTHONPATH`` and
you are done.
4. Check the ``libxml2`` library was installed properly with::
python -c 'import libxml2'
Windows
-------
Download and install:
1. `Twisted for Windows <http://twistedmatrix.com/trac/wiki/Downloads>`_ - you
may need to install `pywin32`_ because of `this Twisted bug`_
2. Install `Zope.Interface`_ (required by Twisted)
3. `libxml2 for Windows <http://users.skynet.be/sbi/libxml-python/>`_
4. `PyOpenSSL for Windows <http://sourceforge.net/project/showfiles.php?group_id=31249>`_
.. _intro-install-step3:
Step 3. Install Scrapy
======================
There are three ways to download and install Scrapy:
1. :ref:`intro-install-release`
2. :ref:`intro-install-easy`
3. :ref:`intro-install-dev`
* :ref:`intro-install-release` (requires installing dependencies separately)
* :ref:`intro-install-easy` (automatically installs dependencies)
* :ref:`intro-install-pip` (automatically installs dependencies)
.. _intro-install-release:
Installing an official release
------------------------------
Download and install an official release
----------------------------------------
Download Scrapy from the `Download page`_. Scrapy is distributed in two ways: a
source code tarball (for Unix and Mac OS X systems) and a Windows installer
(for Windows). If you downloaded the tarball, you can install it as any Python
package using ``setup.py``::
tar zxf scrapy-X.X.X.tar.gz
cd scrapy-X.X.X
tar zxf Scrapy-X.X.X.tar.gz
cd Scrapy-X.X.X
python setup.py install
If you downloaded the Windows installer, just run it.
@ -169,71 +73,105 @@ If you downloaded the Windows installer, just run it.
.. _intro-install-easy:
Installing with `easy_install`_
-------------------------------
Installing with ``easy_install``
--------------------------------
You can install Scrapy running `easy_install`_ like this::
You can install Scrapy using setuptools_'s ``easy_install`` with::
easy_install -U Scrapy
easy_install Scrapy
.. _easy_install: http://peak.telecommunity.com/DevCenter/EasyInstall
.. _intro-install-pip:
.. _intro-install-dev:
Installing with `pip`_
----------------------
Installing the development version
-----------------------------------
You can install Scrapy using `pip`_ with::
.. note:: If you use the development version of Scrapy, you should subscribe
to the mailing lists to get notified of any changes to the API.
pip install Scrapy
.. _intro-install-platforms:
1. Check out the latest development code from the `Mercurial`_ repository (you
need to install `Mercurial_` first)::
Platform specific instructions
==============================
hg clone http://hg.scrapy.org/scrapy scrapy-trunk
Linux
-----
Ubuntu 9.10 or above
~~~~~~~~~~~~~~~~~~~~
If you're running Ubuntu 9.10 (or above), use the official :ref:`Ubuntu
Packages <topics-ubuntu>`, which already solve all dependencies for you and are
continuously updated with the latest bug fixes.
Debian or Ubuntu (9.04 or older)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you're running Debian Linux, run the following command as root::
apt-get install python-twisted python-libxml2 python-pyopenssl python-simplejson
And then follow the instructions in :ref:`intro-install-scrapy`.
Arch Linux
~~~~~~~~~~
If you are running Arch Linux, run the following command as root::
pacman -S twisted libxml2 pyopenssl python-simplejson
And then follow the instructions in :ref:`intro-install-scrapy`.
Other Linux distros
~~~~~~~~~~~~~~~~~~~
The easiest way to install Scrapy in other Linux distros is through
``easy_install``, which will automatically install Twisted and lxml as
dependencies. See :ref:`intro-install-easy`.
Another way would be to install dependencies, if you know the packages in your
distros that meets them. See :ref:`intro-install-requirements`.
Mac OS X
--------
The easiest way to install Scrapy on Mac is through ``easy_install``, which
will automatically install Twisted and lxml as dependencies.
See :ref:`intro-install-easy`.
Windows
-------
There are two ways to install Scrapy in Windows:
* using ``easy_install`` or ``pip`` - see :ref:`intro-install-easy` or
:ref:`intro-install-pip`
* using the Windows installer, but you need to download and install the
dependencies manually:
1. `Twisted for Windows <http://twistedmatrix.com/trac/wiki/Downloads>`_ - you
may need to install `pywin32`_ because of `this Twisted bug`_
2. Install `Zope.Interface`_ (required by Twisted)
3. `libxml2 for Windows <http://users.skynet.be/sbi/libxml-python/>`_
4. `PyOpenSSL for Windows <http://sourceforge.net/project/showfiles.php?group_id=31249>`_
5. Download the Windows installer from the `Downloads page`_ and install it.
.. _Python: http://www.python.org
.. _Twisted: http://twistedmatrix.com
.. _lxml: http://codespeak.net/lxml/
.. _libxml2: http://xmlsoft.org
.. _pywin32: http://sourceforge.net/projects/pywin32/
.. _simplejson: http://pypi.python.org/pypi/simplejson/
.. _Zope.Interface: http://pypi.python.org/pypi/zope.interface#download
.. _this Twisted bug: http://twistedmatrix.com/trac/ticket/3707
.. _pip: http://pypi.python.org/pypi/pip
.. _setuptools: http://pypi.python.org/pypi/setuptools
.. _Mercurial: http://www.selenic.com/mercurial/
2. Add Scrapy to your Python path
If you're on Linux, Mac or any Unix-like system, you can make a symbolic link
to your system ``site-packages`` directory like this::
ln -s /path/to/scrapy-trunk/scrapy SITE-PACKAGES/scrapy
Where ``SITE-PACKAGES`` is the location of your system ``site-packages``
directory. To find this out execute the following::
python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"
Alternatively, you can define your ``PYTHONPATH`` environment variable so that
it includes the ``scrapy-trunk`` directory. This solution also works on Windows
systems, which don't support symbolic links. (Environment variables can be
defined on Windows systems from the `Control Panel`_).
Unix-like example::
PYTHONPATH=/path/to/scrapy-trunk
Windows example (from command line, but you should probably use the `Control
Panel`_)::
set PYTHONPATH=C:\path\to\scrapy-trunk
3. Make the ``scrapy`` command available
On Unix-like systems, create a symbolic link to the file
``scrapy-trunk/bin/scrapy`` in a directory on your system path,
such as ``/usr/local/bin``. For example::
ln -s `pwd`/scrapy-trunk/bin/scrapy /usr/local/bin
This simply lets you type ``scrapy`` from within any directory, rather
than having to qualify the command with the full path to the file.
On Windows systems, the same result can be achieved by copying the file
``scrapy-trunk/bin/scrapy`` to somewhere on your system path,
for example ``C:\Python25\Scripts``, which is customary for Python scripts.
.. _Control Panel: http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/sysdm_advancd_environmnt_addchange_variable.mspx
.. _Downloads page: http://scrapy.org/download/