* Add security warning about pickle files in JOBDIR
The job directory contains files serialized with pickle (spider.state,
request queues), which can execute arbitrary code when loaded. Add a
warning so users know to treat JOBDIR with the same trust level as
their project code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Make JOBDIR security warning more concise and accurate
Simplify the warning to focus on treating the job directory
as trusted, without mentioning specific serialization details.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add Jobdir documentation
* Fix link format
* remove doubtful comments
* some more edits
* Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components
* Minor edits
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Add Pydantic support documentation
* Fix Sphinx references in Pydantic section for non-Sphinx docs
* Minor reformatting
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Initial release notes for 2.14.0, up to ed63fa9.
* Cover 2.14 in the release notes up to 393d715.
* Cover 2.14 in the release notes up to eb49647.
* Cover 2.14 in the release notes up to 426aafd.
* Cover 2.14 in the release notes up to 1e8de24.
* Cover 2.14 in the release notes up to 5a7e132.
* Bump sphinx-lint.
* Finalize the 2.14.0 release notes.
* Drop more of the old versionadded directives.
* Address feedback.
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
* Add a basic Crawler.crawl_async().
* Add custom loop tests for *CrawlerRunner.
* Add AsyncCrawlerProcess.
* Update related docs.
* Update practices.rst.
* Address test failures.
* Add a note about AsyncCrawler* to the docs about switching reactors.
* Address feedback.
* Update for TID253.
* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
* Initial BaseSpiderMiddleware.
* Rename the new methods.
* Remove the spider argument from new BaseSpiderMiddleware methods.
* Add docs for BaseSpiderMiddleware.
* Silence pylint.
* Add BaseSpiderMiddleware tests.
* Add a release note.
* Enable AsyncioSelectorReactor by default.
* Improve get_crawler(), switch more tests to it.
* Fix the remaining default-reactor test failures.
* Address documentation feedback.
* Make pinned envs more consistent.
* Remove BaseDupeFilter.log()
It is never called because request_seen() always returns False
* Document the interface of DUPEFILTER_CLASS classes
* Remove unnecessary BaseDupeFilter comments and add a short class description
* Improve the documentation related to the DUPEFILTER_CLASS setting
* Deprecate BaseDupeFilter.log
* Update the docs
* Fix the new code example
* Remove typing to keep the example short
Otherwise, it would have required yet another import line (from __future__ or typing).
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* new 'allow_offsite' parameter in OffsiteMiddleware
* document deprecated dont_filter flag in OffsiteMiddleware
* avoid deprecating dont_filter in OffsiteMiddleware
* Copy the code to the downloader mw.
* Add tests for allow_offsite in the downloader mw.
* Mark allow_offsite with reqmeta.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Cover 2.12 in the release notes up to 9bb973dc54
* Add one more highlight
* Better merge of the news entries.
* Cover 2.12 in the release notes up to 642af40.
* Cover 2.12 in the release notes up to 7a0a34b.
* Cover 2.12 in the release notes up to b4bad97.
* Add not yet merged PRs #6463, #6507, #6511 to the 2.12 release notes.
* Cover 2.12 in the release notes up to d85c39f, small fixes.
* Cover 2.12 in the release notes up to d215669.
* Cover #6527 in the release notes.
* Address PR feedback.
* Cover recent PRs.
* Finalize the 2.12.0 release notes, small additional fixes.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Documentation added for Spider State in extensions.rst
* Made correction in documentation for Spiderstate
* Added appropriate intro for Spider state extension
* Added reference for spiderstate extension
* Added Spiderstate extension hyperlink refrence in jobs.rst
Kaileshwar R K
Julian Hamze
Adrian Chaves
Yuval Elbar
Anant Murmu
Martin Schimandl
Michael
Andrey Rakhmatullin
Adrián Chaves
Adham 3llam
Thalison Fernandes
Sebastien Dionne
Felipe Benevolo
Laerte Pereira
Keval Sakhiya
Mehraz Hossain Rumman
Laerte Pereira
guillermo-bondonno
anubhav
Kevin Lloyd Bernal
Robert Palmer
Rohit Kumar Singh
mmoriniere
Georgiy Zatserklianyi
mlmsmith