RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,814 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

8814 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 08e4eaf973 Import Deferred directly in scrapy/utils/defer.py. 2021-04-13 22:41:01 +05:00
Andrey Rakhmatullin cef0a8b3d6 Import Deferred directly in scrapy/middleware.py. 2021-04-13 21:08:06 +05:00
Andrey Rahmatullin a8de04c823
Update scrapy/utils/defer.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:30 +05:00
Andrey Rahmatullin b0e7512574
Update scrapy/middleware.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:25 +05:00
Andrey Rahmatullin 335a256752
Update scrapy/core/spidermw.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:20 +05:00
Andrey Rakhmatullin 76fa2257ef Add typing also for return values, other small fixes. 2021-04-13 20:22:02 +05:00
Andrey Rakhmatullin 5b547f0808 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-13 19:47:20 +05:00
Andrey Rahmatullin 15edb3f656
Merge pull request #5095 from wRAR/scraper-slot-cleanup 
Cleanup of slot handling in Scraper.
 2021-04-13 19:46:26 +05:00
Andrey Rahmatullin 38010997a4
Merge pull request #5098 from storymode7/docs-update 
Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation
 2021-04-13 18:52:10 +05:00
Mayank Singhal a4415e4e6f Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation 2021-04-13 17:56:44 +05:30
Adrián Chaves 18502981f7
Merge pull request #5094 from elacuesta/fix-item-class-in-engine-tests 
Engine tests: fix item class spider, add minimal type hints
 2021-04-13 11:07:15 +02:00
Eugenio Lacuesta d8d1dc5b50
Ignore typing warning in test 2021-04-12 10:43:02 -03:00
Andrey Rakhmatullin 4673f05dde Cleanup of slot handling in Scraper. 2021-04-09 23:42:24 +05:00
Eugenio Lacuesta b6f77806b0
Engine tests: fix item class spider, add minimal type hints 2021-04-09 13:09:47 -03:00
Adrián Chaves 5a75b14a5f docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17 2021-04-07 14:04:00 +02:00
Andrey Rakhmatullin e63188cbf7 Bump version: 2.4.1 → 2.5.0 2021-04-06 19:13:32 +05:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Andrey Rakhmatullin 7dc857668f Also some typing for Scraper. 2021-04-04 16:15:33 +05:00
Andrey Rakhmatullin a0101361e2 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-04 16:01:47 +05:00
Andrey Rahmatullin 099fb6ead0
Merge pull request #5076 from wRAR/qualname 
Use __qualname__ in middleware handling.
 2021-04-04 15:52:32 +05:00
Andrey Rakhmatullin 414dd1119a Drop an unused import. 2021-04-03 17:54:55 +05:00
Andrey Rakhmatullin a9e96f9907 Add typing for middleware and coroutine related code. 2021-04-03 17:40:45 +05:00
Andrey Rakhmatullin 9e3b868dd8 Use __qualname__ in middleware handling. 2021-04-03 17:04:09 +05:00
Mikhail Korobov 8c5a3a5189
Merge pull request #5073 from elacuesta/fix-urlencode-doseq-arg 
Fix type for urlencode's doseq argument
 2021-04-02 00:51:15 +05:00
Eugenio Lacuesta 9e7cbc05ae
Fix type for urlencode's doseq argument 2021-04-01 15:22:51 -03:00
Mikhail Korobov 127a8586e1
Merge pull request #5063 from noviluni/update_usageError_message 
Update UsageError message
 2021-04-01 22:47:59 +05:00
Mikhail Korobov 12f2006b7b
Merge pull request #4799 from GeorgeA92/patch-2 
httpcompression stats added
 2021-04-01 22:47:24 +05:00
Mikhail Korobov f0c8d31193
Merge pull request #4694 from Jgaldos/improve-httpstatus-all-meta 
Improve http status all on http error middleware
 2021-04-01 22:43:45 +05:00
Akshay Sharma cc095aa895
Windows pip installation guide (#4736) 
* added initial steps

* fixing link

* python3 -> python

* remaining steps

* steps updated

* Update docs/intro/install.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* added link to Visual Studio

* removed 'install V'

* Update docs/intro/install.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-01 22:39:33 +05:00
Mikhail Korobov 0007a4c117
Merge pull request #4895 from elacuesta/minimal-type-hints 
Some type hints
 2021-04-01 22:36:33 +05:00
Kader DJEHAF ad74567469
Fix argument type (int -> bool) (#4950) 
* Fix warning: Expected type 'bool', got 'int' instead

* Update defer.py

* Fix warning: Expected type 'bool', got 'int' instead

Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-04-01 22:33:56 +05:00
anay2103 5492972d8a
added customized filtering examples in logging.rst (#4965) 
* added customized filtering examples in logging.rst

* Update logging.rst

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-01 22:30:48 +05:00
Mikhail Korobov b09ff99d81
Merge pull request #4974 from Gallaecio/ul-margin 
Move lists closer to their introducing paragraph
 2021-04-01 22:27:59 +05:00
Mikhail Korobov e0a2d2b3c9
Merge pull request #5014 from jpmckinney/patch-1 
docs: Clarify there's one extension instance per crawler
 2021-04-01 22:26:20 +05:00
Mikhail Korobov d0e23486b4
Merge pull request #4902 from Gallaecio/retry-request 
Implement a retry request function
 2021-04-01 22:23:56 +05:00
Eugenio Lacuesta d458ccff3b
Retry request: priority_adjust cannot be float (Request.priority is int) 2021-04-01 12:27:35 -03:00
Mikhail Korobov 9a91586f6f
Merge pull request #5072 from zloidemon/rebranding_update_ga 
Updated GA code for docs, remove Segment tracking
 2021-04-01 20:26:11 +05:00
Veniamin Gvozdikov 90fe494ba2 Rebranding, updated GA code 2021-04-01 11:11:28 +03:00
Eugenio Lacuesta 1d200258a5
Adjust h2 version requirement (#5066) 2021-03-26 14:45:26 +01:00
Eugenio Lacuesta 9c9e1a318d
[HTTP/1.1] Skip Content-Length header if its value is UNKNOWN_LENGTH (#5062) 2021-03-25 15:58:39 +01:00
Pratik Mahankal f0e1a33225
Sort the list of Request.meta alphabetically #5061 (#5065) 2021-03-23 22:16:50 +05:00
Marc 64d4ae1a19 Update UsageError message 2021-03-22 21:46:05 +01:00
Eugenio Lacuesta 72e8cea8af
Avoid exceptions in is_generator_with_return_value (#4935) 2021-03-22 11:51:11 -03:00
Adrián Chaves ec5a7918ec
Include Content-Length in HTTP/1.1 responses (#5057) 2021-03-22 11:25:40 +01:00
Adrián Chaves 2973d8d51a
Remove unnecessary reference to private parsel.Selector._default_type (#5006) 2021-03-22 11:24:10 +01:00
Mikhail Korobov 0e7ee1256c
Merge pull request #5036 from dmiwell/urllength-dont-skip-silently 
UrlLengthMiddleware: don't skip silently
 2021-03-20 00:55:33 +05:00
Adrián Chaves 308a58aa27
Update CI to support Twisted 21.2.0 (#5027) 2021-03-19 18:39:44 +01:00
Adrián Chaves 0dad0fce72
Use pip<20.3 to fix ReadTheDocs builds (#5052) 2021-03-19 11:13:05 +01:00
Adrián Chaves 8e302f2906
Merge pull request #5053 from Gallaecio/constraints 
Fix master CI issues due to pip backtracking
 2021-03-19 10:34:36 +01:00
Adrián Chaves a390b934de Do not install mitmproxy in Python 3.9 2021-03-18 23:53:58 +01:00