RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,775 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

43 Commits

Author SHA1 Message Date
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin 4d31277bc6
Explicitly mark re-exports. (#6579) 2025-01-02 23:48:14 +01:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin 558b1d11d2
Use CallbackT for Request.callback. (#6422) 2024-07-01 09:30:49 +02:00
Andrey Rakhmatullin e47110f9a5
Add parameteres to most Deferred instances. (#6414) 2024-06-26 10:01:43 +02:00
Andrey Rakhmatullin da42e8f124 Add parameters to typing.Dict. 2024-05-31 21:11:50 +05:00
Bagaudin Magomedov b88f22c6c5
Add SpiderLoggerAdapter, change Spider.logger to return SpiderLoggerAdapter (#6324) 2024-05-13 14:33:24 +02:00
Andrey Rakhmatullin b8e333c8ce
Full typing for scrapy/spiders. (#6356) 2024-05-13 12:07:51 +02:00
Andrey Rakhmatullin 93f0628530
Improve typing for Spider.parse(). (#6274) 2024-05-13 11:55:45 +02:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Andrey Rakhmatullin 084a9ba076 Full typing for scrapy/crawler.py and scrapy/spiders/__init__.py. 2023-08-09 23:56:16 +04:00
Andrey Rakhmatullin c43798cb9b More typing for scrapy/utils/defer.py and scrapy/utils/spider.py. 2023-08-06 00:35:33 +04:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Andrey Rahmatullin 45c2bd7d9c
Merge pull request #5150 from gustavo-bordin/master 
changing dunder-str to dunder-repr
 2022-11-23 13:12:06 +05:00
Matthew Donoughe 065db7b566
fix some mypy issues 2022-10-19 08:28:26 -04:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Gustavo Bordin 5bacc4822e changing dunder-str to dunder-repr 2021-05-16 18:40:30 -03:00
Eugenio Lacuesta 91f8144552
Remove deprecated Spider.make_requests_from_url method 2021-04-07 15:16:55 -03:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Eugenio Lacuesta a8114d3731
Typing: annotate a few Spider attributes 2020-08-26 09:00:36 -03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Eugenio Lacuesta 2be2bdd2df
Merge remote-tracking branch 'upstream/master' into spider.parse 2020-07-08 14:00:54 -03:00
Eugenio Lacuesta a915af2e45
Remove monkeypatches module, E402 flake8 code 2020-05-15 20:41:51 -03:00
Eugenio Lacuesta bde0384dfd
Merge branch 'master' into spider.parse 2020-03-14 19:53:36 -03:00
nyov 3b0820d747
Deprecate Spider.make_requests_from_url, part 2 (#4412) 2020-03-12 20:15:49 +01:00
Eugenio Lacuesta 5982e3477c
Take keyword arguments in base parsing methods 2019-12-23 14:12:21 -03:00
nyov b9a58798ee
change Scraper API to call internal `_parse` method 
A Spider class using internal pre-processing can have first dibs
at this and then call a public `parse` method for subclass hooking.
 2019-12-23 13:55:06 -03:00
Mikhail Korobov 250da28952
Merge pull request #4170 from mabelvj/4133-handle-start_url 
Raise error when start_url found instead of start_urls.
 2019-12-05 17:47:03 +05:00
Mabel Villalba 070b3a4e84
Merge branch 'master' into 4133-handle-start_url 2019-11-21 17:10:31 +01:00
Mabel Villalba 1718e450ef [start_url] Fixes #4133: Raise AttributeError error when empty 'start_urls' and 'start_url' found. Added test. 2019-11-21 17:06:47 +01:00
Adrián Chaves a2bf340bab Remove unused imports 2019-11-21 14:21:17 +01:00
nyov 7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
Kiran Koduru 2960c9b568 Use self.__class__.__name__ instead of showing generic Spider class name 2017-08-05 16:29:41 -04:00
kirankoduru be71f98e92 Explicit message for scrapy parse callback 
The scrapy parse method raises a NotImplementedError when not defined,
but for new comers it can be hard to debug what might be going wrong.

Adding an explicit message for NotImplementedError will help new users.
 2017-08-05 15:40:38 -04:00
Mikhail Korobov 5adacc4dde fixed method override check in Python 2 2017-02-17 02:35:02 +05:00
Mikhail Korobov a1e8a8525d fix make_requests_from_url deprcation implementation, add tests 2017-02-17 00:18:29 +05:00
Mikhail Korobov 692975acb4 deprecate Spider.make_requests_from_url. Fixes #1495. 2017-02-16 03:39:34 +05:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina d4926091ab Delete scrapy/spiders/__init__.py 2015-05-09 03:32:01 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00