mirror of https://github.com/scrapy/scrapy.git
c6c9c4efd8
When a crawl with FEED_EXPORT_BATCH_ITEM_COUNT (or FEEDS.batch_item_count) and a %(batch_id) URI template was restarted with the same JOBDIR, the in-memory batch counter reset to 1 and silently overwrote files produced by the previous run. Store the last successfully-written batch_id per uri_template in <JOBDIR>/feedexport.state (JSON) and resume from last+1 on the next run. The state file is only written when JOBDIR is set and the URI template references %(batch_id), so non-JOBDIR crawls and time-only templates behave exactly as before. Adds a regression test that runs the exporter twice against the same JOBDIR and asserts prior batch files are not overwritten. Closes #5153 |
||
|---|---|---|
| .github | ||
| docs | ||
| extras | ||
| scrapy | ||
| sep | ||
| tests | ||
| tests_typing | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .readthedocs.yml | ||
| AUTHORS | ||
| CITATION.cff | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| INSTALL.md | ||
| LICENSE | ||
| NEWS | ||
| README.rst | ||
| SECURITY.md | ||
| codecov.yml | ||
| conftest.py | ||
| pyproject.toml | ||
| tox.ini | ||
README.rst
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
</head>
</html>
Scrapy is a web scraping framework to extract structured data from websites. It is cross-platform, and requires Python 3.10+. It is maintained by Zyte (formerly Scrapinghub) and many other contributors.
Install with:
System Message: WARNING/2 (<stdin>, line 52)
Cannot analyze code. Pygments package not found.
.. code:: bash
pip install scrapy
And follow the documentation to learn how to use it.
If you wish to contribute, see Contributing.