RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,377 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

300 Commits

Author SHA1 Message Date
Magnus Offermanns fd692f3091
Prevent running the -O and -t command-line options together (#5605) 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2022-10-27 14:43:31 +02:00
Mohammadtaher Abbasi 043575123c
Add async callback support to the parse command (#5577) 2022-10-15 10:11:05 +02:00
Adrián Chaves 00ccb02f1f Merge remote-tracking branch 'upstream/2.6' 2022-07-25 14:13:22 +02:00
Andreas Tziortziortziopoulos 83c1939281 Issue #3264, fix error handling when spider is not matched 
Changes
Implementation:
- Check whether Spider exists or is None, and if it's None skip execution of start_requests() with non existing Spider
Testing:
- Add a test case with invalid url inside test_command_parse
  Test proves that non-matched Spider does not throw an AttributeError
 2022-05-06 03:59:30 +03:00
Alex 915c288205 edit 2022-04-24 15:17:48 -07:00
Alex 56c9098d6a changed default depth to 1 2022-04-24 15:17:48 -07:00
Alex b0f5503cb8 Fixes Issue #5481 2022-04-24 15:17:48 -07:00
Alex e2e2ffd0d1
Move from optparse to argparse (#5374) 2022-02-09 20:52:07 +01:00
Andrey Rakhmatullin df04e934e6 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-23 17:25:56 +05:00
Andrey Rakhmatullin 46ef9cf771 Don't install non-working shutdown handlers in `scrapy shell`. 2021-12-23 16:05:17 +05:00
yogender26 d6a384b3cf
corrrection of coma (#5347) 2021-12-22 23:39:05 +01:00
Matsievskiy S.V 3f635eb683
Extract domain from genspider URL (#4439) 2021-08-24 11:05:50 +02:00
Andrey Rahmatullin ada539a63a
Merge pull request #4676 from Gallaecio/startproject-existing-folder 
Fix startproject failing for existing folders
 2021-08-24 12:18:49 +05:00
Eugenio Lacuesta 9e7cbc05ae
Fix type for urlencode's doseq argument 2021-04-01 15:22:51 -03:00
Mikhail Korobov 127a8586e1
Merge pull request #5063 from noviluni/update_usageError_message 
Update UsageError message
 2021-04-01 22:47:59 +05:00
Marc 64d4ae1a19 Update UsageError message 2021-03-22 21:46:05 +01:00
Eugenio Lacuesta ef09e0d10f
Some type hints 2020-12-02 10:39:10 -03:00
Valdir Stumm Junior a2c4a7f920 Add missing f-string prefix to genspider output 2020-11-08 20:14:59 -03:00
Akshay Sharma 159e2b2e2f
allowing to run .pyw files (#4646) 
* allow .pyw in scrapy/commands/runspider.py

* aesthetics

* added tests for '.pyw'

* created class for testing .pyw files

* name=None parameter in get_log

* small fix

* .pyw tests for non-windows

* used @skipIf for tests

* two more tests skipped
 2020-10-01 23:53:08 +05:00
Ammar Najjar 90ca9350f5
Update scrapy/commands/check.py 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-29 08:03:03 +00:00
Ammar Najjar c77450990d
Update scrapy/commands/version.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:41:24 +00:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Ajay Mittur a8e08d51cd
Check if file is already present on running `scrapy genspider` and terminate if so (#4623) 2020-08-17 10:45:52 +02:00
Andrey Rahmatullin 0cf1340c29
Merge pull request #4663 from ajaymittur28/scrapy-check-test 
Add Tests to `scrapy check` command
 2020-08-10 16:18:29 +05:00
Adrián Chaves 1cc8d5829f Remove unneeded try-except 
Exceptions only happen when find_spec gets a 2nd parameter.
 2020-08-06 13:52:47 +02:00
Marc Hernández 336f19f5cc
Change super syntax (#4707) 2020-08-04 20:42:01 +02:00
Adrián Chaves 544c1f6e39 Fix the issue 2020-07-13 16:30:34 +02:00
Andrey Rahmatullin 0ebba175ea
Merge pull request #4666 from Gallaecio/execution-access 
Fix permission handling on project generation from template files
 2020-07-13 16:01:06 +05:00
Andrey Rahmatullin b6f8693db9
Merge pull request #4673 from elacuesta/remove-backslash 
Remove backslash
 2020-07-13 11:14:00 +05:00
Aditya Kumar 3f7e8635f4
Allow the parse command to write data to a file (#4377) 2020-07-11 08:48:24 +02:00
Eugenio Lacuesta 9e99be982a
Remove backslash 2020-07-09 09:51:28 -03:00
Ajay Mittur b6d4f28ea7
Merge branch 'master' into scrapy-check-test 2020-07-08 15:13:37 +05:30
Adrián Chaves ca77ca1f75 Generate read-only files on the fly 2020-07-07 14:44:03 +02:00
Adrián Chaves 79b4dfc53e Fix permission handling on project generation from template files 2020-07-07 14:07:04 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Adrián Chaves 76a2cbf0ff
Apply minor style changes 2020-06-09 21:30:19 +02:00
Matthias Meschede 2df3b54c7d refactor 2020-06-05 09:29:05 +02:00
Matthias Meschede 0cabf406d5 set write permission to startproject folder 2020-06-04 10:39:40 +02:00
Andrey Rahmatullin 63929e75e3
Merge pull request #4552 from jay24rajput/common_commands 
Code sharing between crawl and runspider command
 2020-06-03 12:06:12 +05:00
Jay Rajput 604fe33bad
Update scrapy/commands/__init__.py 
Changed typo in a comment for BaseRunSpiderCommand

Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2020-05-16 01:53:49 +05:30
Jay Rajput cd32a9d9f4 Merge branch 'common_commands' of https://github.com/jay24rajput/scrapy into common_commands 2020-05-16 00:27:02 +05:30
Jay Rajput 4cdd00e21f Changed BaseRunSpiderCommands to BaseRunSpiderCommand 2020-05-16 00:25:57 +05:30
Adrián Chaves 7701c39655
Merge branch 'master' into common_commands 2020-05-15 13:44:45 +02:00
Eugenio Lacuesta ee13c3e95f
Merge branch 'master' into deprecate-baseitem 2020-05-14 13:31:16 -03:00
Jay Rajput 07e125f4c9 Travis CI fixes in __init__.py 2020-05-12 16:31:56 +05:30
Jay Rajput 28d223dd87
Update __init__.py 2020-05-12 15:28:22 +05:30
Jay Rajput e01c30f0d5
Update scrapy/commands/__init__.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-05-12 01:05:20 +05:30
Jay Rajput 02ac6664a5 Travis CI fixes 2020-05-10 00:26:48 +05:30
Jay Rajput ed4f4f8408 Applied suggested format changes 2020-05-10 00:08:34 +05:30