mirror of https://github.com/scrapy/scrapy.git
12dff513d7
Start closing and storing each batch as soon as it reaches the configured item count. Track unfinished close tasks so spider shutdown still waits for all deliveries before emitting the exporter-closed signal. Add an end-to-end regression test that verifies the first batch is stored while the crawl is still running. |
||
|---|---|---|
| .github | ||
| docs | ||
| extras | ||
| scrapy | ||
| sep | ||
| tests | ||
| tests_typing | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .readthedocs.yml | ||
| AUTHORS | ||
| CITATION.cff | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| INSTALL.md | ||
| LICENSE | ||
| NEWS | ||
| README.rst | ||
| SECURITY.md | ||
| codecov.yml | ||
| conftest.py | ||
| pyproject.toml | ||
| tox.ini | ||
README.rst
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
</head>
</html>
Scrapy is a web scraping framework to extract structured data from websites. It is cross-platform, and requires Python 3.10+. It is maintained by Zyte (formerly Scrapinghub) and many other contributors.
Install with:
System Message: WARNING/2 (<stdin>, line 52)
Cannot analyze code. Pygments package not found.
.. code:: bash
pip install scrapy
And follow the documentation to learn how to use it.
If you wish to contribute, see Contributing.