From 0cc1d870dbba316e07a311c7f5803e5f2a9a1e4f Mon Sep 17 00:00:00 2001 From: Rolando Espinoza Date: Mon, 30 Sep 2013 08:43:25 -0400 Subject: [PATCH 1/2] docs: minor tidy up sample code and missing shell prompts. --- docs/topics/djangoitem.rst | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/topics/djangoitem.rst b/docs/topics/djangoitem.rst index 1242bcb97..382c1a8c4 100644 --- a/docs/topics/djangoitem.rst +++ b/docs/topics/djangoitem.rst @@ -27,6 +27,7 @@ Let's see some examples: Creating a Django model for the examples:: from django.db import models + class Person(models.Model): name = models.CharField(max_length=255) age = models.IntegerField() @@ -34,14 +35,15 @@ Creating a Django model for the examples:: Defining a basic :class:`DjangoItem`:: from scrapy.contrib.djangoitem import DjangoItem + class PersonItem(DjangoItem): django_model = Person :class:`DjangoItem` work just like :class:`~scrapy.item.Item`:: - p = PersonItem() - p['name'] = 'John' - p['age'] = '22' + >>> p = PersonItem() + >>> p['name'] = 'John' + >>> p['age'] = '22' To obtain the Django model from the item, we call the extra method :meth:`~DjangoItem.save` of the :class:`DjangoItem`:: @@ -72,10 +74,12 @@ As said before, we can add other fields to the item:: django_model = Person sex = Field() - p = PersonItem() - p['name'] = 'John' - p['age'] = '22' - p['sex'] = 'M' +:: + + >>> p = PersonItem() + >>> p['name'] = 'John' + >>> p['age'] = '22' + >>> p['sex'] = 'M' .. note:: fields added to the item won't be taken into account when doing a :meth:`~DjangoItem.save` From d6e3eae5279c69e8417188ea5a8b27317b5f7843 Mon Sep 17 00:00:00 2001 From: Rolando Espinoza Date: Mon, 30 Sep 2013 09:47:07 -0400 Subject: [PATCH 2/2] docs: added section regarding setting up django's settings. --- docs/topics/djangoitem.rst | 44 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/docs/topics/djangoitem.rst b/docs/topics/djangoitem.rst index 382c1a8c4..5be75c9e1 100644 --- a/docs/topics/djangoitem.rst +++ b/docs/topics/djangoitem.rst @@ -100,3 +100,47 @@ models, but bear in mind that Django ORM may not scale well if you scrape a lot of items (ie. millions) with Scrapy. This is because a relational backend is often not a good choice for a write intensive application (such as a web crawler), specially if the database is highly normalized and with many indices. + +Django settings set up +====================== + +To use the Django models outside the Django application you need to set up the +``DJANGO_SETTINGS_MODULE`` environment variable and --in most cases-- modify +the ``PYTHONPATH`` environment variable to be able to import the settings +module. + +There are many ways to do this depending on your use case and preferences. +Below is detailed one of the simplest ways to do it. + +Suppose your Django project is named ``mysite``, is located in the path +``/home/projects/mysite`` and you have created an app ``myapp`` with the model +``Person``. That means your directory structure is something like this:: + + /home/projects/mysite + ├── manage.py + ├── myapp + │   ├── __init__.py + │   ├── models.py + │   ├── tests.py + │   └── views.py + └── mysite + ├── __init__.py + ├── settings.py + ├── urls.py + └── wsgi.py + +Then you need to add ``/home/projects/mysite`` to the ``PYTHONPATH`` +environment variable and set up the environment variable +``DJANGO_SETTINGS_MODULE`` to ``mysite.settings``. That can be done in your +Scrapy's settings file by adding the lines below:: + + import sys + sys.path.append('/home/projects/mysite') + + import os + os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings' + +Notice that we modify the ``sys.path`` variable instead the ``PYTHONPATH`` +environment variable as we are already within the python runtime. If everything +is right, you should be able to start the ``scrapy shell`` command and import +the model ``Person`` (i.e. ``from myapp.models import Person``).