RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

50 Commits

Author SHA1 Message Date
Andrey Rakhmatullin b68330811b
Deprecate spider arguments of StatsCollector methods (#7011) 2025-08-20 09:52:22 +02:00
Andrey Rakhmatullin e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. (#6824) 
* Don't use CrawlerProcess in the commands that don't need it.

* Use a dummy spider loader in runspider.
 2025-05-28 19:57:33 +05:00
Andrey Rakhmatullin 05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 
* Add a basic Crawler.crawl_async().

* Add custom loop tests for *CrawlerRunner.

* Add AsyncCrawlerProcess.

* Update related docs.

* Update practices.rst.

* Address test failures.

* Add a note about AsyncCrawler* to the docs about switching reactors.

* Address feedback.

* Update for TID253.

* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
 2025-05-28 18:55:44 +05:00
Andrey Rakhmatullin 82acef3051
Add AsyncCrawlerRunner. (#6796) 2025-05-14 16:21:18 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Adrián Chaves 10089c6fe2
2.12 release notes (#6226) 
* Cover 2.12 in the release notes up to 9bb973dc54

* Add one more highlight

* Better merge of the news entries.

* Cover 2.12 in the release notes up to 642af40.

* Cover 2.12 in the release notes up to 7a0a34b.

* Cover 2.12 in the release notes up to b4bad97.

* Add not yet merged PRs #6463, #6507, #6511 to the 2.12 release notes.

* Cover 2.12 in the release notes up to d85c39f, small fixes.

* Cover 2.12 in the release notes up to d215669.

* Cover #6527 in the release notes.

* Address PR feedback.

* Cover recent PRs.

* Finalize the 2.12.0 release notes, small additional fixes.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-11-18 13:07:32 +05:00
Andrey Rakhmatullin d3b5c9be97 Deprecate running Crawler.crawl() twice. 2023-09-06 19:29:44 +04:00
Andrey Rakhmatullin 3ba2dc4d68 Fixes for addon docs. 2023-07-24 17:49:04 +04:00
Andrey Rakhmatullin a2264d3b8b Improve docs about setting settings in addons. 2023-07-14 18:57:27 +04:00
Andrey Rakhmatullin 898e3045a1 Merge branch 'master' into addons 2023-06-14 15:37:08 +04:00
pankaj1707k 03f32c018f
Update black version and fix minor typos and punctuations 2023-02-03 11:52:20 +05:30
pankaj1707k cc9eb3fa79
Format python console blocks using blacken-docs 
Prepend '.. code-block:: pycon' to make python console blocks detectable by blacken-docs
 2023-02-01 20:37:39 +05:30
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Adrián Chaves 3f0a677c04
Cover version directive usage in the documentation policy (#4310) 
* Cover version directives in the documentation policy

* Remove version directives in preparation for Scrapy 2.0

* Update the policy based on the deprecation policy

* Only remove version directives after 3 years
 2020-08-27 23:56:58 +05:00
Rohit Sharma 0cc0e51ca3 remove uneeded escape sequence 
removed uneeded wscape sequences from method arguments in the docs
folder
 2020-05-24 13:38:44 -07:00
Adrián Chaves e84cb18ca0 Use InterSphinx to link to the Twisted documentation 2019-11-18 15:50:45 +01:00
Adrián Chaves d7795484dc Document scrapy.crawler.Crawler.stop() 2019-06-25 17:29:49 +02:00
Capi Etheriel 7d36fa7435
Fix documentation for spiderloader 2019-05-24 10:32:55 -03:00
Adrián Chaves 75d6f56c8a Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
Rolando Espinoza 6431e7a138 DOC State explicitly that spiders are loaded recursively. 2016-12-01 13:26:19 -03:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Jakob de Maeyer d8af395d76 Introduce add-ons via AddonManager and Addon base class 2015-11-11 21:08:19 +01:00
Jakob de Maeyer bb6dee611c Move Settings documentation to docstrings 2015-10-27 12:43:33 +01:00
Jakob de Maeyer a769a1ef78 Introduce BaseSettings with full dictionary interface 2015-10-27 12:38:52 +01:00
Mikhail Korobov 6c9daf3a95 DOC remove unnecessary links; fix references in send_catch_log_deferred docstring 2015-06-10 01:44:19 +05:00
Mikhail Korobov 1740fcf1a6 DOC SignalManager docstrings. See GH-713. 
This change is not 100% backwards compatible because of *args changes.
Their usage was not documented, so we're not breaking public interface.
 2015-06-08 21:05:58 +05:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina 3a71504d35 Extend CrawlerProcess documentation 2015-04-29 23:09:20 -03:00
Julia Medina a2ce78db63 Use autoclass for CrawlerRunner in docs/topics/api.rst 2015-04-29 23:09:20 -03:00
Julia Medina b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Wieland Hoffmann de6501ed1b Remove a duplicate word 2015-04-29 22:31:48 +02:00
Mikhail Korobov 403e7c7c70 rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
Julia Medina 3dabde6706 Update docs for CrawlerRunner.crawl() new usage 2015-04-10 12:09:07 -03:00
Julia Medina 86b0951327 Delete `crawler_deferreds` doc in CrawlerRunner 
This attribute is now an internal one since it's main use-case was
covered by CrawlerRunner.stop().
 2015-04-10 12:06:27 -03:00
Julia Medina 16e62e9c9b Per-spider settings documentation 2014-09-01 21:56:57 -03:00
Julia Medina 419026615f Deprecate Crawler.spiders attribute 2014-08-14 09:19:41 -03:00
Julia Medina d40273561d CrawlerProcess cleanup changes 2014-08-12 14:02:55 -03:00
Julia Medina 980e30a187 Crawler interface cleanup 2014-08-12 14:02:55 -03:00
Julia Medina d7038b2a13 SpiderManager interface cleanup 2014-08-12 14:02:55 -03:00
Julia Medina 39c6a80f9d Both getdict and getlist return copies of the requested values 2014-08-12 14:02:55 -03:00
Julia Medina 3ae971468f Add Settings.copy, freeze and frozencopy method 2014-08-12 14:02:55 -03:00
Julia Medina ed033854e2 setmodule helper method on Settings class 2014-06-10 11:26:50 -03:00
Julia Medina 00e6470ed0 Fix and extend the documentation of the new Settings api 2014-06-10 11:26:38 -03:00
Pablo Hoffman 2a5c7ed4da make Crawler.start() return a deferred that is fired when the crawl is finished 2013-03-20 14:48:59 -03:00
Chris Tilden aae6aed4fb fixes spelling errors in documentation 2013-01-22 14:52:18 -08:00
Pablo Hoffman 81ed2d2d0b major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (stats_spider_opened, etc). Stats are much simpler now, backwards compatibility is kept on the Stats Collector API. 2012-09-14 12:31:33 -03:00
Pablo Hoffman babfc6e79b Updated documentation after singleton removal changes. 
Also removed some unused code and made some minor additional
refactoring.
 2012-08-28 18:35:57 -03:00