RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,890 Commits 42 Branches 132 Tags 162 MiB
Commit Graph

840 Commits

Author SHA1 Message Date
Eugenio Lacuesta c92e8ad0a5
Merge branch 'master' into request-types 2021-06-01 11:01:23 -03:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Eugenio Lacuesta 19c7415aae
Request type hints 2021-05-01 16:34:46 -03:00
Eugenio Lacuesta ddea6b7bfa
Scheduler: minimal interface, API docs (#3559) 2021-04-26 16:16:14 -03:00
Eugenio Lacuesta e779ed7d93
Dupefilter type hints (#5108) 2021-04-20 21:39:07 +02:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Andrey Rakhmatullin 08e4eaf973 Import Deferred directly in scrapy/utils/defer.py. 2021-04-13 22:41:01 +05:00
Andrey Rahmatullin a8de04c823
Update scrapy/utils/defer.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:30 +05:00
Andrey Rakhmatullin 76fa2257ef Add typing also for return values, other small fixes. 2021-04-13 20:22:02 +05:00
Andrey Rakhmatullin a9e96f9907 Add typing for middleware and coroutine related code. 2021-04-03 17:40:45 +05:00
Mikhail Korobov 0007a4c117
Merge pull request #4895 from elacuesta/minimal-type-hints 
Some type hints
 2021-04-01 22:36:33 +05:00
Eugenio Lacuesta 72e8cea8af
Avoid exceptions in is_generator_with_return_value (#4935) 2021-03-22 11:51:11 -03:00
Adrián Chaves 5b2d3e17a3
Merge branch 'master' into http2 2021-03-09 14:59:17 +01:00
Djiar 3894ebb149 Refactor curl_to_request_kwargs #5001 
Co-authored-by: alkazaz alkazaz@kth.se
Co-authored-by: swill swill@kth.se
Co-authored-by: lerjevik lerjevik@kth.se
Co-authored-by: aljica aljica@kth.se
 2021-02-24 11:04:48 +01:00
Eugenio Lacuesta d698b5147b
Merge branch 'master' into http2 2020-12-31 11:13:25 -03:00
Kader DJEHAF 44a7ab5bf0
Fix warning: Expected type 'bool', got 'int' instead (#4940) 
* Fix warning: Expected type 'bool', got 'int' instead

* Update defer.py
 2020-12-30 11:22:27 -03:00
Eugenio Lacuesta ef09e0d10f
Some type hints 2020-12-02 10:39:10 -03:00
Eugenio Lacuesta 6091f3cc03
Remove unused pickle import 2020-12-01 10:26:21 -03:00
Eugenio Lacuesta a80bafe5cd
Remove deprecated SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE 2020-11-30 19:03:13 -03:00
Adrián Chaves 487e19528b
Merge pull request #4901 from elacuesta/remove_deprecated_stuff 
Remove deprecated stuff
 2020-11-30 14:41:40 +01:00
Eugenio Lacuesta fe8bb6bd90
Fix import 2020-11-23 16:51:04 -03:00
Eugenio Lacuesta 0a93df9efd
Move collect_asyncgen to utils.asyncgen 2020-11-23 16:16:18 -03:00
Eugenio Lacuesta 51ca4d0138
Remove deprecated scrapy.utils.gz.is_gzipped function 2020-11-23 15:47:08 -03:00
Eugenio Lacuesta 4075e1eadd
Remove deprecated modules (utils.http/markup/multipart) 2020-11-23 15:07:56 -03:00
Eugenio Lacuesta 075ab156b0
Deprecate scrapy.utils.py36 module 2020-11-23 11:59:59 -03:00
Eugenio Lacuesta 5b5478ae9d
Call asyncio.get_event_loop when installing the asyncio reactor 2020-11-05 14:01:34 -03:00
Adrián Chaves 269fe35d6d
Merge branch 'master' into http2 2020-10-06 21:05:10 +02:00
Andrey Rahmatullin b1255b016a
Merge pull request #4746 from Gallaecio/fix_iternodes 
Fix iternodes
 2020-10-06 20:14:04 +05:00
Andrey Rahmatullin 9b1f86b613
Use f-strings 2020-10-06 18:50:55 +05:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00
Adrián Chaves c3b740f078 Use mocking for tests/test_pipeline_files.py::TestS3FilesStore::test_persist 2020-09-21 23:25:37 +02:00
Andrey Rahmatullin 5e997587d9
Remove dead boto2 code, deprecate is_botocore() (#4776) 2020-09-20 15:06:46 +02:00
Adrián Chaves 9731d91f34
Merge branch 'master' into fix_iternodes 2020-09-01 12:38:37 +02:00
Adrián Chaves 8a3ba34a75 Merge remote-tracking branch 'upstream/master' into http2 2020-08-31 16:43:32 +02:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Andrey Rahmatullin 59a0157ef1
Merge pull request #4722 from Gallaecio/umask 
Do not let umask affect the permissions of startproject-generated files
 2020-08-28 18:41:02 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Ammar Najjar 5ab1a318e8
test: list appears in ValueError Exception message 
Issue #4324
 2020-08-26 15:11:46 +02:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
Ammar Najjar 9aaddcde0a
Update scrapy/utils/conf.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:44:20 +00:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Iván de Prado 0524df8669 Code simplification. 
Thanks @victor-torres for the suggestion
 2020-08-25 14:37:10 +01:00
Iván de Prado 3e726b9df7 Support for delegated methods as callbacks 
It can be useful to structure the spiders code around some helper classes.
 2020-08-25 11:31:20 +01:00
Aditya 26d344b762 Merge branch 'http2' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-24 14:58:23 +05:30
Adrián Chaves afd3a4d116 Fix style issue 2020-08-21 17:04:02 +02:00
Adrián Chaves d711eca4d9 Merge remote-tracking branch 'upstream/master' into fix_iternodes 2020-08-21 13:52:52 +02:00