* Add security warning about pickle files in JOBDIR
The job directory contains files serialized with pickle (spider.state,
request queues), which can execute arbitrary code when loaded. Add a
warning so users know to treat JOBDIR with the same trust level as
their project code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Make JOBDIR security warning more concise and accurate
Simplify the warning to focus on treating the job directory
as trusted, without mentioning specific serialization details.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Make test_start_deprecated_super() more robust.
* Work around the FTPFilesStore.FTP_* initialization.
* Enable xdist on CI.
* Add PYTEST_ADDOPTS to tox passenv.
* Fix test for getwithbase() to ensure class-keyed overrides with None are handled correctly
* Test actual import paths from default_settings
* Improvements
* Remove unused logger
* Run pre-commit
* Restore and improve duplicate key handling and warning
* type → object
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Add Jobdir documentation
* Fix link format
* remove doubtful comments
* some more edits
* Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components
* Minor edits
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Add Pydantic support documentation
* Fix Sphinx references in Pydantic section for non-Sphinx docs
* Minor reformatting
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
* Await some missed deferreds.
* Simplify test_engine_stop_download_*.
* Fix the header value.
* Extend TestHttpWithCrawlerBase.
* Make test_tls_logging more universal.
* Add more tests for header handling.
* Clarify certificate and ip_address integration tests.
* Make test_download_with_maxsize_very_large_file() more universal.
* Fix test_coroutine_asyncio().
* Typo.
* Foundations for the reactorless mode.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
Adrian Chaves
Andrey Rakhmatullin
Yuval Elbar
pierreeurope
Anant Murmu
Varun Chawla
Ryotaro
Martin Schimandl
Michael
Albert Eduardovich N.
Pádraic Slattery
Adrián Chaves
Eugenio Lacuesta
Eugenio Lacuesta
Santiago Beltrán Ruiz