scrapy

Commit Graph

Author	SHA1	Message	Date
Andrey Rakhmatullin	d8ba1571e7	Assorted docs fixes, part 2. (#7725 ) * Assorted docs fixes, part 2. * Second pass. * Address feedback.	2026-07-14 00:49:44 +05:00
Labib Bin Salam	e0a7de7213	Document cb_kwargs/meta deep-copy when JOBDIR is set (#7573 ) When JOBDIR is enabled, requests are serialized to disk with pickle, so the objects stored in a request's cb_kwargs and meta are deep-copied on the round trip. Callbacks then receive copies rather than the original objects, which is easy to miss and can silently break code that relies on sharing mutable state. Add a note to the request serialization section of the jobs docs and a cross-referenced caution to the Request.cb_kwargs attribute docs. Closes #6120	2026-06-11 14:40:15 +05:00
Yuval Elbar	da1a6b7ebc	Add security warning about pickle files in JOBDIR (#7274 ) * Add security warning about pickle files in JOBDIR The job directory contains files serialized with pickle (spider.state, request queues), which can execute arbitrary code when loaded. Add a warning so users know to treat JOBDIR with the same trust level as their project code. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Make JOBDIR security warning more concise and accurate Simplify the warning to focus on treating the job directory as trusted, without mentioning specific serialization details. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-20 19:36:26 +05:00
Martin Schimandl	66fe5de139	Add Jobdir documentation (#5260 ) * Add Jobdir documentation * Fix link format * remove doubtful comments * some more edits * Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components * Minor edits --------- Co-authored-by: Adrian Chaves <adrian@zyte.com>	2026-02-06 22:33:02 +04:00
Adrián Chaves	036f3e5627	Support asynchronous start requests (#6729 )	2025-05-07 19:04:03 +02:00
Rohit Kumar Singh	7701e590fb	Documentation added for Spider State in extensions.rst (#6522 ) * Documentation added for Spider State in extensions.rst * Made correction in documentation for Spiderstate * Added appropriate intro for Spider state extension * Added reference for spiderstate extension * Added Spiderstate extension hyperlink refrence in jobs.rst	2024-11-02 10:45:27 +05:00
pankaj1707k	c1bbb299d7	Add and run pre-commit hook 'blacken-docs' Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.	2023-02-01 16:30:57 +05:30
kamran890	e5998fb846	Document spider.state attribute (#5174 )	2021-09-22 03:15:12 +05:00
Marlena Chatzigrigoriou	d7deba7e89	Document all import paths and use the shortest in examples (#5099 )	2021-07-14 10:34:28 +02:00
madeny	eff96038c7	Correct some typos This won't be an issue if your spider doesn't rely on cookies.	2020-09-26 22:50:38 +02:00
Adrián Chaves	6aa0ba4553	Write release notes for Scrapy 2.0.0 (#4329 )	2020-03-03 09:11:11 +01:00
Akshay Sharma	182445f9d9	Fix a spelling error: ie. → i.e. (#4338 )	2020-02-18 17:58:31 +01:00
Marc Hernandez Cabot	c0d84f0962	fix typos	2019-12-18 19:39:21 +01:00
Adrián Chaves	aef98188fa	Improve the details about request serialization requirements for JOBDIR	2019-11-07 18:06:55 +01:00
Eugenio Lacuesta	8fb077694f	Request.cb_kwargs: Update docs	2019-03-28 16:32:35 -03:00
Adrián Chaves	75d6f56c8a	Switch from ` to `` where inline code formatting is desired	2019-03-01 16:56:58 +01:00
Harry Moreno	b828b5f8c8	fix grammar	2019-01-26 18:39:05 -05:00
Mikhail Korobov	0ccead6681	DOC more Python 3 in examples	2018-09-12 17:57:27 +05:00
Aditya	9dd680d5c9	Use https for external links wherever possible in docs	2017-10-26 23:33:45 +05:30
darshanime	d8e62e660d	update log demo print	2016-07-26 20:46:12 +05:30
darshanime	0c77b6d033	update docs for settings	2016-07-25 18:05:51 +05:30
Darshan Chaudhary	472a8a47d0	Change name, log once	2016-07-25 18:03:43 +05:30
Ramiro Morales	933dbc6be6	Oops	2015-03-25 18:33:17 -03:00
Ramiro Morales	ca2575001e	Add missing callback arg in jobs topic example.	2015-03-25 18:32:20 -03:00
Mikhail Korobov	2d3803672b	DOC use top-level shortcuts in docs	2014-04-15 01:09:35 +06:00
Chris Tilden	aae6aed4fb	fixes spelling errors in documentation	2013-01-22 14:52:18 -08:00
Natan L	d572f8945e	Fixed typo 'persitent' --> 'persistent'	2012-12-31 11:14:01 -08:00
Pablo Hoffman	40f7075f11	added initial documentation about suspend and resume crawls	2011-09-02 13:12:27 -03:00

28 Commits