RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

9291 Commits

Author SHA1 Message Date
Eugenio Lacuesta 91f8144552
Remove deprecated Spider.make_requests_from_url method 2021-04-07 15:16:55 -03:00
Adrián Chaves 5a75b14a5f docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17 2021-04-07 14:04:00 +02:00
Ricardo Amendoeira 8603f9d7a5
Apply changes to other examples in the same section. 2021-04-06 20:23:07 +01:00
Andrey Rakhmatullin e63188cbf7 Bump version: 2.4.1 → 2.5.0 2021-04-06 19:13:32 +05:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Mayank Singhal e7d51886ef Find bash from PATH instead of /bin/bash 2021-04-06 02:21:18 +05:30
Andrey Rakhmatullin 7dc857668f Also some typing for Scraper. 2021-04-04 16:15:33 +05:00
Andrey Rakhmatullin a0101361e2 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-04 16:01:47 +05:00
Andrey Rahmatullin 099fb6ead0
Merge pull request #5076 from wRAR/qualname 
Use __qualname__ in middleware handling.
 2021-04-04 15:52:32 +05:00
Andrey Rakhmatullin 414dd1119a Drop an unused import. 2021-04-03 17:54:55 +05:00
Andrey Rakhmatullin a9e96f9907 Add typing for middleware and coroutine related code. 2021-04-03 17:40:45 +05:00
Andrey Rakhmatullin 9e3b868dd8 Use __qualname__ in middleware handling. 2021-04-03 17:04:09 +05:00
Andrey Rakhmatullin 7bd1d888d4 More robust sync/async middleware mix checking. 2021-04-02 23:07:24 +05:00
Andrey Rakhmatullin 30ed7fa349 Some cleanup, make sync middlewares fail earlier. 2021-04-02 22:20:56 +05:00
Andrey Rakhmatullin 849472535e Update docs. 2021-04-02 20:20:35 +05:00
Mikhail Korobov 8c5a3a5189
Merge pull request #5073 from elacuesta/fix-urlencode-doseq-arg 
Fix type for urlencode's doseq argument
 2021-04-02 00:51:15 +05:00
Eugenio Lacuesta 9e7cbc05ae
Fix type for urlencode's doseq argument 2021-04-01 15:22:51 -03:00
Mikhail Korobov 127a8586e1
Merge pull request #5063 from noviluni/update_usageError_message 
Update UsageError message
 2021-04-01 22:47:59 +05:00
Mikhail Korobov 12f2006b7b
Merge pull request #4799 from GeorgeA92/patch-2 
httpcompression stats added
 2021-04-01 22:47:24 +05:00
Mikhail Korobov f0c8d31193
Merge pull request #4694 from Jgaldos/improve-httpstatus-all-meta 
Improve http status all on http error middleware
 2021-04-01 22:43:45 +05:00
Akshay Sharma cc095aa895
Windows pip installation guide (#4736) 
* added initial steps

* fixing link

* python3 -> python

* remaining steps

* steps updated

* Update docs/intro/install.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* added link to Visual Studio

* removed 'install V'

* Update docs/intro/install.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-01 22:39:33 +05:00
Mikhail Korobov 0007a4c117
Merge pull request #4895 from elacuesta/minimal-type-hints 
Some type hints
 2021-04-01 22:36:33 +05:00
Kader DJEHAF ad74567469
Fix argument type (int -> bool) (#4950) 
* Fix warning: Expected type 'bool', got 'int' instead

* Update defer.py

* Fix warning: Expected type 'bool', got 'int' instead

Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-04-01 22:33:56 +05:00
anay2103 5492972d8a
added customized filtering examples in logging.rst (#4965) 
* added customized filtering examples in logging.rst

* Update logging.rst

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-01 22:30:48 +05:00
Mikhail Korobov b09ff99d81
Merge pull request #4974 from Gallaecio/ul-margin 
Move lists closer to their introducing paragraph
 2021-04-01 22:27:59 +05:00
Mikhail Korobov e0a2d2b3c9
Merge pull request #5014 from jpmckinney/patch-1 
docs: Clarify there's one extension instance per crawler
 2021-04-01 22:26:20 +05:00
Mikhail Korobov d0e23486b4
Merge pull request #4902 from Gallaecio/retry-request 
Implement a retry request function
 2021-04-01 22:23:56 +05:00
Eugenio Lacuesta d458ccff3b
Retry request: priority_adjust cannot be float (Request.priority is int) 2021-04-01 12:27:35 -03:00
Mikhail Korobov 9a91586f6f
Merge pull request #5072 from zloidemon/rebranding_update_ga 
Updated GA code for docs, remove Segment tracking
 2021-04-01 20:26:11 +05:00
Veniamin Gvozdikov 90fe494ba2 Rebranding, updated GA code 2021-04-01 11:11:28 +03:00
Ricardo Amendoeira b247fa9982
Include loading settings in `Running multiple spiders in the same process` section 
The example in the documentation doesn't take into account the project settings
 2021-03-29 01:48:28 +01:00
Andrey Rakhmatullin 6803779eb7 Add more tests for _process_iterable_universal. 2021-03-26 22:29:07 +05:00
Andrey Rakhmatullin b5f501df7b Remove some unneeded code from _AsyncCooperatorAdapter. 2021-03-26 20:17:41 +05:00
Eugenio Lacuesta 1d200258a5
Adjust h2 version requirement (#5066) 2021-03-26 14:45:26 +01:00
Andrey Rakhmatullin 7b515a0485 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-03-25 22:13:49 +05:00
Andrey Rakhmatullin 0638d6f01c Fix handling middlewares that change sync iterables into async. 2021-03-25 21:58:29 +05:00
Andrey Rakhmatullin f422861ef4 Add more tests for spider middlewares. 2021-03-25 20:49:12 +05:00
Andrey Rakhmatullin a97dc55c71 Add/improve docs. 2021-03-25 20:49:12 +05:00
Andrey Rakhmatullin 0596f2bf6e Remove not needed deferred_from_coro call. 2021-03-25 20:49:07 +05:00
Eugenio Lacuesta 9c9e1a318d
[HTTP/1.1] Skip Content-Length header if its value is UNKNOWN_LENGTH (#5062) 2021-03-25 15:58:39 +01:00
Andrey Rakhmatullin a6034f942b Add tests for _AsyncCooperatorAdapter. 2021-03-23 22:47:37 +05:00
Andrey Rakhmatullin c51ec1ae1c Drop process_iterable_helper, add _process_iterable_universal. 2021-03-23 22:47:30 +05:00
Pratik Mahankal f0e1a33225
Sort the list of Request.meta alphabetically #5061 (#5065) 2021-03-23 22:16:50 +05:00
Marc 64d4ae1a19 Update UsageError message 2021-03-22 21:46:05 +01:00
Eugenio Lacuesta 72e8cea8af
Avoid exceptions in is_generator_with_return_value (#4935) 2021-03-22 11:51:11 -03:00
Adrián Chaves ec5a7918ec
Include Content-Length in HTTP/1.1 responses (#5057) 2021-03-22 11:25:40 +01:00
Adrián Chaves 2973d8d51a
Remove unnecessary reference to private parsel.Selector._default_type (#5006) 2021-03-22 11:24:10 +01:00
Mikhail Korobov 0e7ee1256c
Merge pull request #5036 from dmiwell/urllength-dont-skip-silently 
UrlLengthMiddleware: don't skip silently
 2021-03-20 00:55:33 +05:00
Adrián Chaves 308a58aa27
Update CI to support Twisted 21.2.0 (#5027) 2021-03-19 18:39:44 +01:00
Adrián Chaves 0dad0fce72
Use pip<20.3 to fix ReadTheDocs builds (#5052) 2021-03-19 11:13:05 +01:00