mirror of https://github.com/scrapy/scrapy.git
Merge pull request #404 from darkrho/djangoitem-docs
Add a section to DjangoItem docs page regarding setting up Django's settings.
This commit is contained in:
commit
56275b1b9c
|
|
@ -27,6 +27,7 @@ Let's see some examples:
|
|||
Creating a Django model for the examples::
|
||||
|
||||
from django.db import models
|
||||
|
||||
class Person(models.Model):
|
||||
name = models.CharField(max_length=255)
|
||||
age = models.IntegerField()
|
||||
|
|
@ -34,14 +35,15 @@ Creating a Django model for the examples::
|
|||
Defining a basic :class:`DjangoItem`::
|
||||
|
||||
from scrapy.contrib.djangoitem import DjangoItem
|
||||
|
||||
class PersonItem(DjangoItem):
|
||||
django_model = Person
|
||||
|
||||
:class:`DjangoItem` work just like :class:`~scrapy.item.Item`::
|
||||
|
||||
p = PersonItem()
|
||||
p['name'] = 'John'
|
||||
p['age'] = '22'
|
||||
>>> p = PersonItem()
|
||||
>>> p['name'] = 'John'
|
||||
>>> p['age'] = '22'
|
||||
|
||||
To obtain the Django model from the item, we call the extra method
|
||||
:meth:`~DjangoItem.save` of the :class:`DjangoItem`::
|
||||
|
|
@ -72,10 +74,12 @@ As said before, we can add other fields to the item::
|
|||
django_model = Person
|
||||
sex = Field()
|
||||
|
||||
p = PersonItem()
|
||||
p['name'] = 'John'
|
||||
p['age'] = '22'
|
||||
p['sex'] = 'M'
|
||||
::
|
||||
|
||||
>>> p = PersonItem()
|
||||
>>> p['name'] = 'John'
|
||||
>>> p['age'] = '22'
|
||||
>>> p['sex'] = 'M'
|
||||
|
||||
.. note:: fields added to the item won't be taken into account when doing a :meth:`~DjangoItem.save`
|
||||
|
||||
|
|
@ -96,3 +100,47 @@ models, but bear in mind that Django ORM may not scale well if you scrape a lot
|
|||
of items (ie. millions) with Scrapy. This is because a relational backend is
|
||||
often not a good choice for a write intensive application (such as a web
|
||||
crawler), specially if the database is highly normalized and with many indices.
|
||||
|
||||
Django settings set up
|
||||
======================
|
||||
|
||||
To use the Django models outside the Django application you need to set up the
|
||||
``DJANGO_SETTINGS_MODULE`` environment variable and --in most cases-- modify
|
||||
the ``PYTHONPATH`` environment variable to be able to import the settings
|
||||
module.
|
||||
|
||||
There are many ways to do this depending on your use case and preferences.
|
||||
Below is detailed one of the simplest ways to do it.
|
||||
|
||||
Suppose your Django project is named ``mysite``, is located in the path
|
||||
``/home/projects/mysite`` and you have created an app ``myapp`` with the model
|
||||
``Person``. That means your directory structure is something like this::
|
||||
|
||||
/home/projects/mysite
|
||||
├── manage.py
|
||||
├── myapp
|
||||
│ ├── __init__.py
|
||||
│ ├── models.py
|
||||
│ ├── tests.py
|
||||
│ └── views.py
|
||||
└── mysite
|
||||
├── __init__.py
|
||||
├── settings.py
|
||||
├── urls.py
|
||||
└── wsgi.py
|
||||
|
||||
Then you need to add ``/home/projects/mysite`` to the ``PYTHONPATH``
|
||||
environment variable and set up the environment variable
|
||||
``DJANGO_SETTINGS_MODULE`` to ``mysite.settings``. That can be done in your
|
||||
Scrapy's settings file by adding the lines below::
|
||||
|
||||
import sys
|
||||
sys.path.append('/home/projects/mysite')
|
||||
|
||||
import os
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
|
||||
|
||||
Notice that we modify the ``sys.path`` variable instead the ``PYTHONPATH``
|
||||
environment variable as we are already within the python runtime. If everything
|
||||
is right, you should be able to start the ``scrapy shell`` command and import
|
||||
the model ``Person`` (i.e. ``from myapp.models import Person``).
|
||||
|
|
|
|||
Loading…
Reference in New Issue