RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,093 Commits 49 Branches 132 Tags 145 MiB
Commit Graph

10093 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 1a0572ad02 Reinstall the log handler after loading per-spider settings. 2023-09-12 16:46:35 +04:00
Andrey Rakhmatullin bb15c93a2b Add Crawler._load_settings(). 2023-09-12 14:44:16 +04:00
Andrey Rakhmatullin 6629a61dd9 Fix one more testcase. 2023-09-08 20:15:49 +04:00
Andrey Rakhmatullin 036d5836d0 Move more things from Crawler.__init__() to .crawl(). 2023-09-08 19:29:05 +04:00
Andrey Rakhmatullin 97b98bf181 Roll back the update_settings() changes. 2023-09-07 17:02:37 +04:00
Andrey Rakhmatullin b4380995da Merge remote-tracking branch 'origin/master' into change-init-order 2023-09-06 20:34:27 +04:00
Andrey Rakhmatullin 00527fdcbe
Merge pull request #6040 from wRAR/deprecate-double-crawl 
Deprecate running Crawler.crawl() twice.
 2023-09-06 20:32:14 +04:00
Andrey Rakhmatullin b39d2d4353 Add a test. 2023-09-06 19:44:22 +04:00
Andrey Rakhmatullin d3b5c9be97 Deprecate running Crawler.crawl() twice. 2023-09-06 19:29:44 +04:00
Andrey Rakhmatullin df112a3996 Move reactor installation into Crawler.crawl(). 2023-09-05 19:43:08 +04:00
Andrey Rakhmatullin 69282c5318 Merge branch 'master' into change-init-order 2023-09-05 19:23:36 +04:00
Víctor Ruiz c1dd5493ac
Fix enabled addons not being logged (#6036) 2023-09-04 08:03:35 +02:00
Andrey Rakhmatullin 276bce0641
Update type hints for Twisted 23.8.0. (#6034) 2023-09-01 08:47:23 +02:00
Andrey Rakhmatullin d9efb94b61 Merge branch '2.10' 2023-08-31 13:21:22 +04:00
Mikhail Korobov 0fcb0554d1
Merge pull request #6016 from jxlil/fix/response.json 
Fix: response.json() call makes unnecessary memory allocation
 2023-08-31 14:07:24 +05:00
Andrey Rakhmatullin cddb8c15d6
Merge pull request #5926 from GeorgeA92/periodic_log_2 
periodic_log: implemented as separate extension
 2023-08-30 22:29:54 +04:00
Andrey Rakhmatullin a320e5f6a4 Bump version: 2.10.0 → 2.10.1 2023-08-30 12:08:36 +04:00
Andrey Rakhmatullin 99c826ce4d
Merge pull request #6027 from wRAR/relnotes-2.10.1 
Release notes for 2.10.1.
 2023-08-30 12:04:48 +04:00
Andrey Rakhmatullin 3fc7726e65
Merge pull request #6026 from wRAR/restrict-twisted 
Forbid Twisted 23.8.0+
 2023-08-29 18:26:19 +04:00
Andrey Rakhmatullin f8f550120b Release notes for 2.10.1. 2023-08-29 18:07:15 +04:00
Andrey Rakhmatullin 9a72a2550c Forbid Twisted 23.8.0+. 2023-08-29 17:00:28 +04:00
Andrey Rakhmatullin 0e7fc60b89
Merge pull request #6021 from dveselov/master-1 
Add .webp to link extractor IGNORED_EXTENSIONS list
 2023-08-21 22:04:03 +04:00
Jalil SA 4dd32672ed added entry to Backward-incompatible changes 2023-08-21 08:28:08 -06:00
Georgiy Zatserklianyi fe02642980 Merge remote-tracking branch 'origin/periodic_log_2' into periodic_log_2 2023-08-21 17:15:42 +03:00
Georgiy Zatserklianyi 7e542846e4 periodic_log: `datetime.utcnow()` changed to `datetime.now(tz=timezone.utc)` 2023-08-21 17:13:18 +03:00
Georgiy Zatserklianyi 1f03cb1419 Merge remote-tracking branch 'upstream/master' into periodic_log_2 2023-08-21 17:07:18 +03:00
Laerte Pereira df2163ce6a
Remove datetime.utcnow() usage (#6014) 2023-08-21 15:51:49 +02:00
Dima Veselov 7355741c7d
Add .webp to IGNORED_EXTENSIONS 2023-08-20 22:38:56 +03:00
Jalil SA fd4292b722 fix: test_cache_json_response 2023-08-17 10:24:34 -06:00
Jalil SA 19867659f3 fix: response.json() call makes unnecessary memory allocation 2023-08-17 10:22:17 -06:00
Andrey Rakhmatullin 3318971512
Merge pull request #6013 from Laerte/master 
Handle Tuple type on getdictorlist method, bump 3.12 python version
 2023-08-14 19:55:38 +04:00
Laerte Pereira b06936f111 Handle Tuple type on getdictorlist method, bump 3.12 python version 2023-08-14 10:33:48 -03:00
Georgiy Zatserklianyi ac1694a9ad
Update docs/topics/extensions.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-08-11 18:49:09 +03:00
Georgiy Zatserklianyi d67be20b2d
Update docs/topics/extensions.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-08-11 18:48:58 +03:00
Georgiy Zatserklianyi 3a4a949f9d
Update docs/topics/extensions.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-08-11 18:48:48 +03:00
Georgiy Zatserklianyi e6bd9829bd
Update docs/topics/extensions.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-08-11 18:48:31 +03:00
Georgiy Zatserklianyi 2f094a7a5c
Update docs/topics/extensions.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-08-11 18:48:14 +03:00
Andrey Rakhmatullin 7c497688f8
Merge pull request #6005 from wRAR/typing-crawler 
More typing for scrapy/*.py
 2023-08-11 14:41:47 +04:00
Adrián Chaves 8a0108b19c
Merge branch 'master' into typing-crawler 2023-08-11 10:42:45 +02:00
Andrey Rakhmatullin 34d050cfe5
Remove deprecated CrawlerRunner.spiders. (#6010) 2023-08-11 10:41:05 +02:00
Andrey Rakhmatullin 44b15c3004 Remove typing for CrawlerRunner.spider_loader. 2023-08-10 23:17:06 +04:00
Andrey Rakhmatullin 9df67a554e Add RequestFingerprinterProtocol. 2023-08-10 22:53:22 +04:00
Georgiy Zatserklianyi 736a4b615c
Update scrapy/settings/default_settings.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-08-10 21:34:35 +03:00
Georgiy Zatserklianyi f05657e542 periodic_log: interval check moved 2023-08-10 21:31:10 +03:00
Laerte Pereira 9e74748fca
Remove extra spider parameter in item pipeline docs (#6009) 2023-08-10 13:48:43 +02:00
Laerte Pereira 4a090d951a
Remove deprecated PythonItemExporter.binary (#6007) 2023-08-10 07:36:42 +02:00
Andrey Rakhmatullin b50268c100 Full typing for scrapy/link.py. 2023-08-09 23:56:16 +04:00
Andrey Rakhmatullin e6d919497d More typing for scrapy/core/scheduler.py. 2023-08-09 23:56:16 +04:00
Andrey Rakhmatullin 7dca18e2e7 Full typing for scrapy/responsetypes.py. 2023-08-09 23:56:16 +04:00
Andrey Rakhmatullin c5885fc13b Full typing for scrapy/exceptions.py. 2023-08-09 23:56:16 +04:00