c6c9c4efd8
When a crawl with FEED_EXPORT_BATCH_ITEM_COUNT (or FEEDS.batch_item_count) and a %(batch_id) URI template was restarted with the same JOBDIR, the in-memory batch counter reset to 1 and silently overwrote files produced by the previous run. Store the last successfully-written batch_id per uri_template in <JOBDIR>/feedexport.state (JSON) and resume from last+1 on the next run. The state file is only written when JOBDIR is set and the URI template references %(batch_id), so non-JOBDIR crawls and time-only templates behave exactly as before. Adds a regression test that runs the exporter twice against the same JOBDIR and asserts prior batch files are not overwritten. Closes #5153 |
||
|---|---|---|
| .. | ||
| _ext | ||
| _static | ||
| _templates | ||
| _tests | ||
| intro | ||
| topics | ||
| utils | ||
| Makefile | ||
| README.rst | ||
| conf.py | ||
| conftest.py | ||
| contributing.rst | ||
| faq.rst | ||
| index.rst | ||
| news.rst | ||
| requirements.in | ||
| requirements.txt | ||
| versioning.rst | ||
README.rst
| orphan: |
|---|
Scrapy documentation quick start guide
This file provides a quick guide on how to compile the Scrapy documentation.
Setup the environment
To compile the documentation you need Sphinx Python library. To install it and all its dependencies run the following command from this dir
pip install -r requirements.txt
Compile the documentation
To compile the documentation (to classic HTML output) run the following command from this dir:
make html
Documentation will be generated (in HTML format) inside the build/html dir.
View the documentation
To view the documentation run the following command:
make htmlview
This command will fire up your default browser and open the main page of your (previously generated) HTML documentation.
Start over
To clean up all generated documentation files and start from scratch run:
make clean
Keep in mind that this command won't touch any documentation source files.
Recreating documentation on the fly
There is a way to recreate the doc automatically when you make changes, you need to install watchdog (pip install watchdog) and then use:
make watch
Alternative method using tox
To compile the documentation to HTML run the following command:
tox -e docs
Documentation will be generated inside the docs/_build/all dir.