From 290ebee090a5d886d825f1373efb7d861c4f2124 Mon Sep 17 00:00:00 2001 From: Julia Medina Date: Fri, 10 Jul 2015 17:14:50 -0300 Subject: [PATCH 1/7] Add OS X installation section to docs Closes #1342 and (possibly) #1126 --- docs/intro/install.rst | 63 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 67507016d..26631ac1f 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -86,6 +86,67 @@ You can follow the generic instructions or install Scrapy from `AUR Scrapy packa yaourt -S scrapy +Mac OS X +-------- + +Building Scrapy's dependencies requires the presence of a C compiler and +development headers. On OS X this is typically provided by Apple’s Xcode +development tools. To install the Xcode command line tools open a terminal +window and run:: + + xcode-select --install + +There's a `known issue `_ that +prevents ``pip`` from updating system packages. This has to be addressed to +successfully install Scrapy and its dependencies. Here are some proposed +solutions: + +* *(Recommended)* **Don't** use system python, install a new, updated version + that doesn't conflict with the rest of your system. Here's how to do it using + the `homebrew`_ package manager: + + * Install `homebrew`_:: + + ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" + + * Update your ``PATH`` variable to state that homebrew packages should be + used before system packages (Change ``.bashrc`` to ``.zshrc`` accordantly + if you're using `zsh`_ as default shell):: + + echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc + + * Reload ``.bashrc`` to ensure the changes have taken place:: + + source ~/.bashrc + + * Install python:: + + brew install python + + * Latest versions of python have ``pip`` bundled with them so you won't need + to install it separately. If this is not the case, upgrade python:: + + brew update; brew upgrade python + +* *(Alternative)* Force system python to load the user installed packages + first: + + * Update your ``PYTHONPATH`` variable (Change ``.bashrc`` to ``.zshrc`` + accordantly if you're using `zsh`_ as default shell):: + + echo "export PYTHONPATH=/Library/Python/2.7/site-packages:$PYTHONPATH" >> ~/.bashrc + + * Reload ``.bashrc`` to ensure the changes have taken place:: + + source ~/.bashrc + + * Install ``pip``:: + + sudo easy_install pip + +After any of these workarounds you should be able to install Scrapy:: + + pip install Scrapy .. _Python: https://www.python.org/ .. _pip: https://pip.pypa.io/en/latest/installing.html @@ -95,3 +156,5 @@ You can follow the generic instructions or install Scrapy from `AUR Scrapy packa .. _OpenSSL: https://pypi.python.org/pypi/pyOpenSSL .. _setuptools: https://pypi.python.org/pypi/setuptools .. _AUR Scrapy package: https://aur.archlinux.org/packages/scrapy/ +.. _homebrew: http://brew.sh/ +.. _zsh: http://www.zsh.org/ From 5723e6b457312465a136a94ba915fb2ffed005ce Mon Sep 17 00:00:00 2001 From: Julia Medina Date: Fri, 10 Jul 2015 18:52:01 -0300 Subject: [PATCH 2/7] Add non-python dependencies to Ubuntu install section in the docs Closes #1314 and closes #1198. --- docs/intro/install.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 26631ac1f..f1ab90db4 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -79,6 +79,16 @@ Instead, use the official :ref:`Ubuntu Packages `, which already solve all dependencies for you and are continuously updated with the latest bug fixes. +If you prefer to build the python dependencies locally instead of relying on +system packages you'll need to install their required non-python dependencies +first:: + + sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev + +You can install Scrapy with ``pip`` after that:: + + pip install Scrapy + Archlinux --------- From 439b376d5bb506ab0e5a0cdecd8af6927a6e2fbf Mon Sep 17 00:00:00 2001 From: Julia Medina Date: Fri, 10 Jul 2015 19:06:23 -0300 Subject: [PATCH 3/7] Note in install docs about pip being already included in python>=2.7.9 --- docs/intro/install.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index f1ab90db4..246964d3f 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -14,7 +14,8 @@ The installation steps assume that you have the following things installed: * `Python`_ 2.7 * `pip`_ and `setuptools`_ Python packages. Nowadays `pip`_ requires and - installs `setuptools`_ if not installed. + installs `setuptools`_ if not installed. Python 2.7.9 and later include + `pip`_ by default, so you may have it already. * `lxml`_. Most Linux distributions ships prepackaged versions of lxml. Otherwise refer to http://lxml.de/installation.html @@ -58,7 +59,8 @@ Windows Be sure you download the architecture (win32 or amd64) that matches your system -* Install `pip`_ from https://pip.pypa.io/en/latest/installing.html +* *(Only required for Python<2.7.9)* Install `pip`_ from + https://pip.pypa.io/en/latest/installing.html Now open a Command prompt to check ``pip`` is installed correctly:: From 320d3a59b9d89b8f3e7c13add04e28e7453a3f96 Mon Sep 17 00:00:00 2001 From: Julia Medina Date: Fri, 10 Jul 2015 19:15:14 -0300 Subject: [PATCH 4/7] Add oldest supported tox version to contributing docs Better fix for #1337 --- docs/contributing.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/contributing.rst b/docs/contributing.rst index ad9a3805a..87cea2a5f 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -146,6 +146,14 @@ tests requires `tox`_. Running tests ------------- +Make sure you have a recent enough `tox`_ installation: + + ``tox --version`` + +If your version is older than 1.7.0, please update it first: + + ``pip install -U tox`` + To run all tests go to the root directory of Scrapy source code and run: ``tox`` From 46ee1c6ee2084caa797da4b2cd441d7551be2dbe Mon Sep 17 00:00:00 2001 From: Julia Medina Date: Wed, 19 Aug 2015 02:21:08 -0300 Subject: [PATCH 5/7] Reference Homebrew's homepage for installation instructions --- docs/intro/install.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 246964d3f..918197e43 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -117,9 +117,7 @@ solutions: that doesn't conflict with the rest of your system. Here's how to do it using the `homebrew`_ package manager: - * Install `homebrew`_:: - - ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" + * Install `homebrew`_ following the instructions in http://brew.sh/ * Update your ``PATH`` variable to state that homebrew packages should be used before system packages (Change ``.bashrc`` to ``.zshrc`` accordantly From d3c3408cd46fd735ea6b2770f7d30c9e2b1ab04b Mon Sep 17 00:00:00 2001 From: Julia Medina Date: Wed, 19 Aug 2015 03:19:27 -0300 Subject: [PATCH 6/7] Replace alternative OSX install workaround with virtualenv --- docs/intro/install.rst | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 918197e43..978d89920 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -138,21 +138,14 @@ solutions: brew update; brew upgrade python -* *(Alternative)* Force system python to load the user installed packages - first: +* *(Optional)* Install Scrapy inside an isolated python environment. - * Update your ``PYTHONPATH`` variable (Change ``.bashrc`` to ``.zshrc`` - accordantly if you're using `zsh`_ as default shell):: + This method is a workaround for the above OS X issue, but it's an overall + good practice for managing dependencies and can complement the first method. - echo "export PYTHONPATH=/Library/Python/2.7/site-packages:$PYTHONPATH" >> ~/.bashrc - - * Reload ``.bashrc`` to ensure the changes have taken place:: - - source ~/.bashrc - - * Install ``pip``:: - - sudo easy_install pip + `virtualenv`_ is a tool you can use to create virtual environments in python. + We recommended reading a tutorial like + http://docs.python-guide.org/en/latest/dev/virtualenvs/ to get started. After any of these workarounds you should be able to install Scrapy:: @@ -168,3 +161,4 @@ After any of these workarounds you should be able to install Scrapy:: .. _AUR Scrapy package: https://aur.archlinux.org/packages/scrapy/ .. _homebrew: http://brew.sh/ .. _zsh: http://www.zsh.org/ +.. _virtualenv: https://virtualenv.pypa.io/en/latest/ From eb7b9d506f599f821842448a1b1dc7b3928decde Mon Sep 17 00:00:00 2001 From: Julia Medina Date: Tue, 8 Sep 2015 18:49:44 -0300 Subject: [PATCH 7/7] Add note to ubuntu install section about debian compatibility --- docs/intro/install.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 978d89920..3adb4e6b0 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -91,6 +91,11 @@ You can install Scrapy with ``pip`` after that:: pip install Scrapy +.. note:: + + The same non-python dependencies can be used to install Scrapy in Debian + Wheezy (7.0) and above. + Archlinux ---------