RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,469 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

57 Commits

Author SHA1 Message Date
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Mohammadtaher Abbasi 043575123c
Add async callback support to the parse command (#5577) 2022-10-15 10:11:05 +02:00
Adrián Chaves 00ccb02f1f Merge remote-tracking branch 'upstream/2.6' 2022-07-25 14:13:22 +02:00
Andreas Tziortziortziopoulos 83c1939281 Issue #3264, fix error handling when spider is not matched 
Changes
Implementation:
- Check whether Spider exists or is None, and if it's None skip execution of start_requests() with non existing Spider
Testing:
- Add a test case with invalid url inside test_command_parse
  Test proves that non-matched Spider does not throw an AttributeError
 2022-05-06 03:59:30 +03:00
Alex 915c288205 edit 2022-04-24 15:17:48 -07:00
Alex 56c9098d6a changed default depth to 1 2022-04-24 15:17:48 -07:00
Alex b0f5503cb8 Fixes Issue #5481 2022-04-24 15:17:48 -07:00
Alex e2e2ffd0d1
Move from optparse to argparse (#5374) 2022-02-09 20:52:07 +01:00
Eugenio Lacuesta ef09e0d10f
Some type hints 2020-12-02 10:39:10 -03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Aditya Kumar 3f7e8635f4
Allow the parse command to write data to a file (#4377) 2020-07-11 08:48:24 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta ee13c3e95f
Merge branch 'master' into deprecate-baseitem 2020-05-14 13:31:16 -03:00
Eugenio Lacuesta 5256eae60d
Meta class to handle isinstance checks for BaseItem 2020-05-07 14:37:41 -03:00
Eugenio Lacuesta 2af34873cc
Flake8: Remove E128 (commands module) 2020-05-06 17:25:14 -03:00
Eugenio Lacuesta c887fe37ad
Simplify parse command 2020-04-04 22:15:36 -03:00
Eugenio Lacuesta 4270e0a0da
Fix E731: do not assign a lambda expression 2020-04-04 21:54:55 -03:00
Marc Hernandez Cabot 6fb85951ce fix E22X flake8 2020-02-21 08:39:14 +01:00
Eugenio Lacuesta 415526d922
Remove __future__ imports 2019-11-02 23:26:38 -03:00
s-sanjay 2c14692e60 remove .keys() to avoid creating a tmp list/keyview obj (#4031) 
Also add --verbose and --nolinks for code coverage
 2019-09-27 09:56:43 +02:00
Eugenio Lacuesta 0522fe35c3 parse command: improve option description 2019-03-29 16:15:34 -03:00
Eugenio Lacuesta f5e0b6b89a parse command: rename cb_kwargs option to cbkwargs 2019-03-29 14:03:26 -03:00
Eugenio Lacuesta e8af6331b5 Add cb_kwargs option to the parse command 2019-03-28 15:03:40 -03:00
IAlwaysBeCoding 846fd83512 removed commented out code, wrapped line to pep-8 and removed backlashes 2017-11-11 18:30:01 -05:00
Valdir Stumm Junior 0f6f486769 fix parse command issue with callback as a string 2017-06-05 16:19:00 -03:00
Joakim Uddholm 743a0aa422 Two fixes for when using the parse command and the '-r' flag (rules). 
1. Use default "parse" as callback when the matching rule has no callback.
2. Log error and return when no rule matches the parsed url.
 2016-09-08 21:52:14 +02:00
Joakim Uddholm 625c69fdc7 Fixed typo in error message when selecting a callback method for the parse command. 2016-08-08 14:32:53 +02:00
Mikhail Korobov 215905bdb6 PY3 port bench, startproject, genspider, list and runspider commands 2015-10-12 17:59:49 +05:00
Julia Medina 616aec92c8 Move scrapy/command.py to scrapy/commands/__init__.py 2015-05-09 03:20:36 -03:00
Julia Medina 4f54ca3294 Change 'scrapy' logger for '__name__' on every module 2015-04-22 17:24:41 -03:00
Julia Medina 7a958f90be Replace scrapy.log calls for their equivalents in the logging std module 
Changes:
 - Each module takes 'scrapy' logger and logs through it
 - Lazy string evaluation in all log messages
 - Added missing log messages in scrapy/core/engine.py
 - Contextual data such as crawler or spider instances, and failures
 2015-04-22 17:24:39 -03:00
Mikhail Korobov ad587ea792 rename CrawlerRunner.spiders to CrawlerRunner.spider_loader 2015-04-17 01:54:26 +05:00
Mikhail Korobov 39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Julia Medina 9cbbfd8b04 Adjust spiders' utils to new SpiderManager API 2014-08-12 14:02:56 -03:00
alexanderlukanin13 ae4a61a2d5 Added "from __future__ import print_function" in all relevant places except xlib 2013-10-20 11:55:07 +06:00
alexanderlukanin13 d381a35732 Python 3 compatible syntax: print, except, raise, octal numbers; removed Python 2.2 boolean compatibility code in xlib/pydispatch/dispatcher.py 2013-10-20 01:59:35 +06:00
Alex Cepoi a812def59e port all scrapy commands to new CrawlerProcess 2013-08-19 23:40:23 +02:00
Nicolás Ramírez 2b39527f72 pipelines argument added 2013-04-08 14:55:28 -03:00
Nicolás Ramírez df19693ed2 Spider Arguments support for parse command and test 2013-03-28 16:49:06 -03:00
Pablo Hoffman dabb064293 fix bug in scrapy parse command when spider is not specified explicitly. closes #209 2012-12-20 11:59:07 -02:00
Daniel Graña dcef7b03c1 format log lines lazily in case they are dropped by loglevels 2012-08-31 17:14:35 -03:00
Alex Cepoi 0350bed8fd parse command: fix crash in case of bad callback 2012-08-31 16:43:04 +02:00
Alexandru Cepoi 2e05cf5685 fix small bug with parse command 2012-06-21 20:06:50 +02:00
Alexandru Cepoi 3b5cf31198 add --verbose option to parse command 2012-05-24 16:35:09 +02:00
Alexandru Cepoi 8e3c5f1bf7 add --depth field to parse command 2012-05-24 15:10:03 +02:00
Pablo Hoffman 84f518fc5e More core changes: 
* removed execution queue (replaced by newer spider queues)
* added real support for returning iterators in Spider.start_requests()
* removed support for passing urls to 'scrapy crawl' command
 2011-07-15 15:18:39 -03:00
Pablo Hoffman 18d303b5f1 ported internal scrapy.utils imports to w3lib 2011-04-19 01:33:52 -03:00
Pablo Hoffman 1b766877f1 Added ISpiderManager interface and a test to verify the default SpiderManager comforms to it 2010-09-03 14:29:27 -03:00
Pablo Hoffman 59f09c50e4 Yet another scrapy.cmdline code refactoring by removing --settings and --version options, adding a version command and adding a UsageError exception for signaling usage errors. Updated all commands accordingly 2010-08-28 18:06:51 -03:00