RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 134 MiB
Commit Graph

338 Commits

Author SHA1 Message Date
guillermo-bondonno 9bc0029d27
Allow updating pre-crawler settings from add-ons (#6568) 2025-01-22 12:07:44 +01:00
anubhav ca345a3b73
Flexible severity of logging level when items are dropped (#6608) 2025-01-15 11:08:18 +01:00
Adrián Chaves 98ba61256d
Deprecate BaseDupeFilter.log() and improve dupefilter docs (#4151) 
* Remove BaseDupeFilter.log()

It is never called because request_seen() always returns False

* Document the interface of DUPEFILTER_CLASS classes

* Remove unnecessary BaseDupeFilter comments and add a short class description

* Improve the documentation related to the DUPEFILTER_CLASS setting

* Deprecate BaseDupeFilter.log

* Update the docs

* Fix the new code example

* Remove typing to keep the example short

Otherwise, it would have required yet another import line (from __future__ or typing).

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-14 19:36:56 +05:00
Adrián Chaves 21b9ba717c
Allow customizing logged software versions (#6582) 
Co-authored-by: Grammy Jiang <grammy.jiang@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-12-16 14:46:23 +01:00
Adrián Chaves f57fc454be Replace Slot.throttle with Request.meta['dont_throttle'] 2024-11-05 19:44:30 +01:00
Andrey Rakhmatullin ce5a132f12
Run and fix linkcheck. (#6524) 2024-11-04 11:40:07 +01:00
mmoriniere 46cddc6ecf
Ignore SyntaxError as well when SPIDER_LOADER_WARN_ONLY is set to True (#6484) 2024-10-02 10:04:03 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Adrián Chaves d7581c6b41
Allow disabling the AutoThrottle extension for a given slot (#6246) 2024-03-12 09:44:29 +01:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Adrián Chaves 1087bb7b2e Update the docs 2023-11-23 09:11:14 +01:00
nihilisticneuralnet 04024f1e79
[Solved] JOBDIR= None for when Scheduler initializes disk queue even if JOBDIR is empty string (#6124) 
Co-authored-by: John Doe <johndoe@email.com>
 2023-10-31 17:37:26 +04:00
Andrey Rakhmatullin 2fa768399a Replace the VERSION vars. 2023-09-15 19:19:42 +04:00
Andrey Rakhmatullin 61ef37a594 Docs improvements. 2023-09-13 15:47:41 +04:00
Andrey Rakhmatullin 61e6bfc023 Docs improvements. 2023-09-13 12:35:15 +04:00
Andrey Rakhmatullin a3f22046ef Document changing settings in Spider.from_crawler(). 2023-09-12 20:41:36 +04:00
Andrey Rakhmatullin 7c6aaedda2
Merge pull request #5846 from jxlil/docs/spiders 
Added documentation for Spider.update_settings
 2023-08-02 21:09:44 +04:00
Andrey Rakhmatullin af1be835e4
Apply suggestions from code review 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-08-02 19:46:16 +04:00
Andrey Rakhmatullin 72462a53e2 Add more docs for update_settings(). 2023-08-02 12:32:53 +04:00
Andrey Rakhmatullin 005c8cc5f0 Unify the "add-on" spelling. 2023-07-19 13:15:35 +04:00
Andrey Rakhmatullin a2264d3b8b Improve docs about setting settings in addons. 2023-07-14 18:57:27 +04:00
Andrey Rakhmatullin 27f5f35134 More quick doc fixes. 2023-06-14 18:30:33 +04:00
Andrey Rakhmatullin fdfab17438 Fix docs for the renamed ADDONS setting. 2023-06-14 18:00:18 +04:00
Andrey Rakhmatullin 898e3045a1 Merge branch 'master' into addons 2023-06-14 15:37:08 +04:00
Andrey Rakhmatullin 67bfb304cd Release notes for the current master. 2023-04-21 20:49:51 +04:00
Andrey Rakhmatullin 3659a8c8d9
Merge pull request #5328 from GeorgeA92/per_slot_settings 
Per slot settings
 2023-03-09 22:31:24 +05:00
Georgiy Zatserklianyi 218829b1db per slot settings: typo fixed 2023-03-06 16:25:49 +02:00
Georgiy Zatserklianyi be52fe4f67 Revert "periodic log stats: typo in docs fixed" 
This reverts commit 68ba25cb69.
 2023-03-06 16:24:28 +02:00
Georgiy Zatserklianyi 68ba25cb69 periodic log stats: typo in docs fixed 2023-03-04 22:15:06 +02:00
Oleg 2b3a8f0d69
Fix a typo in the LOG_FORMAT description (#5839) 2023-03-02 10:19:41 +01:00
Georgiy Zatserklianyi 045092e8d7 per_slot_settings(docs): code style fix (from previous pre-commit check) 2023-02-16 17:55:59 +02:00
Georgiy Zatserklianyi 07e1429877 per_slot_settings(docs): docs `:setting` bindings added 2023-02-16 15:14:01 +02:00
Andrey Rakhmatullin 76eba9977b
Skip or fix failing code block tests (#5826) 2023-02-14 09:40:38 +01:00
Georgiy Zatserklianyi 474087be6f per_slot_settings(docs): settings 3 2023-02-09 14:52:28 +02:00
Georgiy Zatserklianyi 5208d436ae per_slot_settings(docs): settings 2 2023-02-09 00:33:21 +02:00
Georgiy Zatserklianyi c3033a54b1 per_slot_settings(docs): settings 1 2023-02-08 23:55:07 +02:00
pankaj1707k c1bbb299d7
Add and run pre-commit hook 'blacken-docs' 
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
 2023-02-01 16:30:57 +05:30
Andrey Rakhmatullin f87d061069
Merge branch 'master' into download-delay-documentation 2023-01-13 00:22:48 +05:00
Yohanna Lisnichuk b63ca6f834 docs: use the right notes directive 2022-12-15 13:20:45 -03:00
Yohanna Lisnichuk 96e526aad8
docs: change DOWNLOAD_DELAY description format 
To be consistent with the other settings' descriptions
 2022-12-15 10:01:55 -03:00
Adrián Chaves 06c8f673af
2.7 release notes (#5680) 
* Fix the display name of documented fingerprinter class methods

* Initial draft for the Scrapy 2.7 release notes

* Update VERSION and PREVIOUS_VERSION references

* Clarify the restrictions lifted for item field output names

* Fix the description of the BOM bug fix

* Fix the note about changes in MIME sniffing

* Fix typo

* Extend highlights

* Fyx typo
 2022-10-17 18:04:29 +05:00
Andrey Rakhmatullin 62cc26e209 Change TWISTED_REACTOR in the default template. 2022-10-14 22:03:54 +06:00
Gábor Lipták 79a4bc3da0
Cleanup METHOD_SSLv3 
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
 2022-09-25 17:01:43 -04:00
Tim B 5f19420211
Documented how settings must be picklable (#5629) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Laerte Pereira <5853172+Laerte@users.noreply.github.com>
 2022-09-21 08:27:27 +02:00
Adrián Chaves 00ccb02f1f Merge remote-tracking branch 'upstream/2.6' 2022-07-25 14:13:22 +02:00
Adrián Chaves 34e4ed72ea Document how DOWNLOAD_DELAY affects per-domain concurrency 2022-06-21 12:46:54 +02:00
Adrián Chaves 4ef71829b2
If TWISTED_REACTOR is None, reuse any pre-installed reactor (#5528) 2022-06-17 10:37:27 +02:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Péter Ferenc Gyarmati 38d2a154ec docs: use https scheme for each quotes.toscrape.com url occurrence 2022-02-06 18:52:15 +01:00
Andrey Rakhmatullin 6eaceec735 Implement docs suggestions. 2021-12-31 20:14:24 +05:00