From ac007802d6571d73510f8b977d00029b2f69be4e Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Wed, 17 Nov 2010 21:32:23 -0200 Subject: [PATCH] Simplified installation guide, including lxml as alternative dependency to libxml2. Closes #280 --- docs/faq.rst | 5 +- docs/intro/install.rst | 288 ++++++++++++++++------------------------- 2 files changed, 116 insertions(+), 177 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index c369cb61c..b401223f7 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -41,8 +41,9 @@ Does Scrapy work with Python 3.0? --------------------------------- No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment, -Scrapy works with Python 2.5, 2.6 and 2.7. See also: -:ref:`faq-python-versions`. +Scrapy works with Python 2.5, 2.6 and 2.7. + +.. seealso:: :ref:`faq-python-versions`. Did Scrapy "steal" X from Django? --------------------------------- diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 1f5d9a3e0..c58ed9cc9 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -4,16 +4,7 @@ Installation guide ================== -This document describes how to install Scrapy on Linux, Windows and Mac OS X -systems and it consists of the following 3 steps: - -* :ref:`intro-install-step1` -* :ref:`intro-install-step2` -* :ref:`intro-install-step3` - -If you use Ubuntu (9.10 or above), you can use the :ref:`Ubuntu packages -` to install Scrapy more easily, skipping these steps. The -Ubuntu packages are also kept updated with latest bug fixes. +This document describes how to install Scrapy on Linux, Windows and Mac OS X. .. _intro-install-requirements: @@ -25,138 +16,51 @@ Requirements * `Twisted`_ 2.5.0, 8.0 or above (Windows users: you'll need to install `Zope.Interface`_ and maybe `pywin32`_ because of `this Twisted bug`_) -* `libxml2`_ (versions prior to 2.6.28 are known to have problems parsing certain malformed HTML, and have also been reported to contain leaks, so 2.6.28 or above is highly recommended) +* `lxml`_ or `libxml2`_ (if using `libxml2`_, version 2.6.28 or above is highly recommended) * `simplejson`_ (not required if using Python 2.6 or above) -.. _Python: http://www.python.org -.. _Twisted: http://twistedmatrix.com -.. _libxml2: http://xmlsoft.org -.. _pywin32: http://sourceforge.net/projects/pywin32/ -.. _simplejson: http://pypi.python.org/pypi/simplejson/ -.. _Zope.Interface: http://pypi.python.org/pypi/zope.interface#download -.. _this Twisted bug: http://twistedmatrix.com/trac/ticket/3707 +* `pyopenssl `_ (for HTTPS support. Optional, + but highly recommended) -Optional: +.. _intro-install-python: -* `pyopenssl `_ (for HTTPS support, highly recommended) +Install Python +============== -.. _intro-install-step1: +First, you need to install Python, if you haven't done so already. -Step 1. Install Python -====================== +Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at +http://www.python.org/download/ -Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at http://www.python.org/download/ +.. seealso:: :ref:`faq-python-versions` .. highlight:: sh -.. _intro-install-step2: +.. _intro-install-scrapy: -Step 2. Install required libraries -================================== +Install Scrapy +============== -The procedure for installing the required third party libraries depends on the -platform and operating system you use. +There are many ways to install Scrapy. Pick the one you feel more comfortable +with. -Ubuntu/Debian -------------- - -If you're running Ubuntu/Debian Linux, run the following command as root:: - - apt-get install python-twisted python-libxml2 - -To install optional libraries:: - - apt-get install python-pyopenssl python-simplejson - -Arch Linux ----------- - -If you are running Arch Linux, run the following command as root:: - - pacman -S twisted libxml2 - -To install optional libraries:: - - pacman -S pyopenssl python-simplejson - -Mac OS X --------- - -First, download `Twisted for Mac`_. - -.. _Twisted for Mac: http://twistedmatrix.com/trac/wiki/Downloads#MacOSX - -Mac OS X ships an ``libxml2`` version too old to be used by Scrapy. Also, by -looking on the web it seems that installing ``libxml2`` on MacOSX is a bit of a -challenge. Here is a way to achieve this, though not acceptable on the long -run: - -1. Fetch the following libxml2 and libxslt packages: - - ftp://xmlsoft.org/libxml2/libxml2-2.7.3.tar.gz - - ftp://xmlsoft.org/libxml2/libxslt-1.1.24.tar.gz - -2. Extract, build and install them both with:: - - ./configure --with-python=/Library/Frameworks/Python.framework/Versions/2.5/ - make - sudo make install - - Replacing ``/Library/Frameworks/Python.framework/Version/2.5/`` with your - current python framework location. - -3. Install libxml2 Python bidings with:: - - cd libxml2-2.7.3/python - sudo make install - - The libraries and modules should be installed in something like - /usr/local/lib/python2.5/site-packages. Add it to your ``PYTHONPATH`` and - you are done. - -4. Check the ``libxml2`` library was installed properly with:: - - python -c 'import libxml2' - -Windows -------- - -Download and install: - -1. `Twisted for Windows `_ - you - may need to install `pywin32`_ because of `this Twisted bug`_ - -2. Install `Zope.Interface`_ (required by Twisted) - -3. `libxml2 for Windows `_ - -4. `PyOpenSSL for Windows `_ - -.. _intro-install-step3: - -Step 3. Install Scrapy -====================== - -There are three ways to download and install Scrapy: - -1. :ref:`intro-install-release` -2. :ref:`intro-install-easy` -3. :ref:`intro-install-dev` +* :ref:`intro-install-release` (requires installing dependencies separately) +* :ref:`intro-install-easy` (automatically installs dependencies) +* :ref:`intro-install-pip` (automatically installs dependencies) .. _intro-install-release: -Installing an official release ------------------------------- +Download and install an official release +---------------------------------------- Download Scrapy from the `Download page`_. Scrapy is distributed in two ways: a source code tarball (for Unix and Mac OS X systems) and a Windows installer (for Windows). If you downloaded the tarball, you can install it as any Python package using ``setup.py``:: - tar zxf scrapy-X.X.X.tar.gz - cd scrapy-X.X.X + tar zxf Scrapy-X.X.X.tar.gz + cd Scrapy-X.X.X python setup.py install If you downloaded the Windows installer, just run it. @@ -169,71 +73,105 @@ If you downloaded the Windows installer, just run it. .. _intro-install-easy: -Installing with `easy_install`_ -------------------------------- +Installing with ``easy_install`` +-------------------------------- -You can install Scrapy running `easy_install`_ like this:: +You can install Scrapy using setuptools_'s ``easy_install`` with:: - easy_install -U Scrapy + easy_install Scrapy -.. _easy_install: http://peak.telecommunity.com/DevCenter/EasyInstall +.. _intro-install-pip: -.. _intro-install-dev: +Installing with `pip`_ +---------------------- -Installing the development version ------------------------------------ +You can install Scrapy using `pip`_ with:: -.. note:: If you use the development version of Scrapy, you should subscribe - to the mailing lists to get notified of any changes to the API. + pip install Scrapy +.. _intro-install-platforms: -1. Check out the latest development code from the `Mercurial`_ repository (you - need to install `Mercurial_` first):: +Platform specific instructions +============================== - hg clone http://hg.scrapy.org/scrapy scrapy-trunk +Linux +----- +Ubuntu 9.10 or above +~~~~~~~~~~~~~~~~~~~~ + +If you're running Ubuntu 9.10 (or above), use the official :ref:`Ubuntu +Packages `, which already solve all dependencies for you and are +continuously updated with the latest bug fixes. + +Debian or Ubuntu (9.04 or older) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you're running Debian Linux, run the following command as root:: + + apt-get install python-twisted python-libxml2 python-pyopenssl python-simplejson + +And then follow the instructions in :ref:`intro-install-scrapy`. + +Arch Linux +~~~~~~~~~~ + +If you are running Arch Linux, run the following command as root:: + + pacman -S twisted libxml2 pyopenssl python-simplejson + +And then follow the instructions in :ref:`intro-install-scrapy`. + +Other Linux distros +~~~~~~~~~~~~~~~~~~~ + +The easiest way to install Scrapy in other Linux distros is through +``easy_install``, which will automatically install Twisted and lxml as +dependencies. See :ref:`intro-install-easy`. + +Another way would be to install dependencies, if you know the packages in your +distros that meets them. See :ref:`intro-install-requirements`. + +Mac OS X +-------- + +The easiest way to install Scrapy on Mac is through ``easy_install``, which +will automatically install Twisted and lxml as dependencies. + +See :ref:`intro-install-easy`. + +Windows +------- + +There are two ways to install Scrapy in Windows: + +* using ``easy_install`` or ``pip`` - see :ref:`intro-install-easy` or + :ref:`intro-install-pip` + +* using the Windows installer, but you need to download and install the + dependencies manually: + + 1. `Twisted for Windows `_ - you + may need to install `pywin32`_ because of `this Twisted bug`_ + + 2. Install `Zope.Interface`_ (required by Twisted) + + 3. `libxml2 for Windows `_ + + 4. `PyOpenSSL for Windows `_ + + 5. Download the Windows installer from the `Downloads page`_ and install it. + +.. _Python: http://www.python.org +.. _Twisted: http://twistedmatrix.com +.. _lxml: http://codespeak.net/lxml/ +.. _libxml2: http://xmlsoft.org +.. _pywin32: http://sourceforge.net/projects/pywin32/ +.. _simplejson: http://pypi.python.org/pypi/simplejson/ +.. _Zope.Interface: http://pypi.python.org/pypi/zope.interface#download +.. _this Twisted bug: http://twistedmatrix.com/trac/ticket/3707 +.. _pip: http://pypi.python.org/pypi/pip +.. _setuptools: http://pypi.python.org/pypi/setuptools .. _Mercurial: http://www.selenic.com/mercurial/ - -2. Add Scrapy to your Python path - - If you're on Linux, Mac or any Unix-like system, you can make a symbolic link - to your system ``site-packages`` directory like this:: - - ln -s /path/to/scrapy-trunk/scrapy SITE-PACKAGES/scrapy - - Where ``SITE-PACKAGES`` is the location of your system ``site-packages`` - directory. To find this out execute the following:: - - python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()" - - Alternatively, you can define your ``PYTHONPATH`` environment variable so that - it includes the ``scrapy-trunk`` directory. This solution also works on Windows - systems, which don't support symbolic links. (Environment variables can be - defined on Windows systems from the `Control Panel`_). - - Unix-like example:: - - PYTHONPATH=/path/to/scrapy-trunk - - Windows example (from command line, but you should probably use the `Control - Panel`_):: - - set PYTHONPATH=C:\path\to\scrapy-trunk - -3. Make the ``scrapy`` command available - - On Unix-like systems, create a symbolic link to the file - ``scrapy-trunk/bin/scrapy`` in a directory on your system path, - such as ``/usr/local/bin``. For example:: - - ln -s `pwd`/scrapy-trunk/bin/scrapy /usr/local/bin - - This simply lets you type ``scrapy`` from within any directory, rather - than having to qualify the command with the full path to the file. - - On Windows systems, the same result can be achieved by copying the file - ``scrapy-trunk/bin/scrapy`` to somewhere on your system path, - for example ``C:\Python25\Scripts``, which is customary for Python scripts. - .. _Control Panel: http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/sysdm_advancd_environmnt_addchange_variable.mspx - +.. _Downloads page: http://scrapy.org/download/