RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,795 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

8795 Commits

Author SHA1 Message Date
Andrey Rakhmatullin e63188cbf7 Bump version: 2.4.1 → 2.5.0 2021-04-06 19:13:32 +05:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Andrey Rahmatullin 099fb6ead0
Merge pull request #5076 from wRAR/qualname 
Use __qualname__ in middleware handling.
 2021-04-04 15:52:32 +05:00
Andrey Rakhmatullin 9e3b868dd8 Use __qualname__ in middleware handling. 2021-04-03 17:04:09 +05:00
Mikhail Korobov 8c5a3a5189
Merge pull request #5073 from elacuesta/fix-urlencode-doseq-arg 
Fix type for urlencode's doseq argument
 2021-04-02 00:51:15 +05:00
Eugenio Lacuesta 9e7cbc05ae
Fix type for urlencode's doseq argument 2021-04-01 15:22:51 -03:00
Mikhail Korobov 127a8586e1
Merge pull request #5063 from noviluni/update_usageError_message 
Update UsageError message
 2021-04-01 22:47:59 +05:00
Mikhail Korobov 12f2006b7b
Merge pull request #4799 from GeorgeA92/patch-2 
httpcompression stats added
 2021-04-01 22:47:24 +05:00
Mikhail Korobov f0c8d31193
Merge pull request #4694 from Jgaldos/improve-httpstatus-all-meta 
Improve http status all on http error middleware
 2021-04-01 22:43:45 +05:00
Akshay Sharma cc095aa895
Windows pip installation guide (#4736) 
* added initial steps

* fixing link

* python3 -> python

* remaining steps

* steps updated

* Update docs/intro/install.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* added link to Visual Studio

* removed 'install V'

* Update docs/intro/install.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-01 22:39:33 +05:00
Mikhail Korobov 0007a4c117
Merge pull request #4895 from elacuesta/minimal-type-hints 
Some type hints
 2021-04-01 22:36:33 +05:00
Kader DJEHAF ad74567469
Fix argument type (int -> bool) (#4950) 
* Fix warning: Expected type 'bool', got 'int' instead

* Update defer.py

* Fix warning: Expected type 'bool', got 'int' instead

Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-04-01 22:33:56 +05:00
anay2103 5492972d8a
added customized filtering examples in logging.rst (#4965) 
* added customized filtering examples in logging.rst

* Update logging.rst

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-01 22:30:48 +05:00
Mikhail Korobov b09ff99d81
Merge pull request #4974 from Gallaecio/ul-margin 
Move lists closer to their introducing paragraph
 2021-04-01 22:27:59 +05:00
Mikhail Korobov e0a2d2b3c9
Merge pull request #5014 from jpmckinney/patch-1 
docs: Clarify there's one extension instance per crawler
 2021-04-01 22:26:20 +05:00
Mikhail Korobov d0e23486b4
Merge pull request #4902 from Gallaecio/retry-request 
Implement a retry request function
 2021-04-01 22:23:56 +05:00
Eugenio Lacuesta d458ccff3b
Retry request: priority_adjust cannot be float (Request.priority is int) 2021-04-01 12:27:35 -03:00
Mikhail Korobov 9a91586f6f
Merge pull request #5072 from zloidemon/rebranding_update_ga 
Updated GA code for docs, remove Segment tracking
 2021-04-01 20:26:11 +05:00
Veniamin Gvozdikov 90fe494ba2 Rebranding, updated GA code 2021-04-01 11:11:28 +03:00
Eugenio Lacuesta 1d200258a5
Adjust h2 version requirement (#5066) 2021-03-26 14:45:26 +01:00
Eugenio Lacuesta 9c9e1a318d
[HTTP/1.1] Skip Content-Length header if its value is UNKNOWN_LENGTH (#5062) 2021-03-25 15:58:39 +01:00
Pratik Mahankal f0e1a33225
Sort the list of Request.meta alphabetically #5061 (#5065) 2021-03-23 22:16:50 +05:00
Marc 64d4ae1a19 Update UsageError message 2021-03-22 21:46:05 +01:00
Eugenio Lacuesta 72e8cea8af
Avoid exceptions in is_generator_with_return_value (#4935) 2021-03-22 11:51:11 -03:00
Adrián Chaves ec5a7918ec
Include Content-Length in HTTP/1.1 responses (#5057) 2021-03-22 11:25:40 +01:00
Adrián Chaves 2973d8d51a
Remove unnecessary reference to private parsel.Selector._default_type (#5006) 2021-03-22 11:24:10 +01:00
Mikhail Korobov 0e7ee1256c
Merge pull request #5036 from dmiwell/urllength-dont-skip-silently 
UrlLengthMiddleware: don't skip silently
 2021-03-20 00:55:33 +05:00
Adrián Chaves 308a58aa27
Update CI to support Twisted 21.2.0 (#5027) 2021-03-19 18:39:44 +01:00
Adrián Chaves 0dad0fce72
Use pip<20.3 to fix ReadTheDocs builds (#5052) 2021-03-19 11:13:05 +01:00
Adrián Chaves 8e302f2906
Merge pull request #5053 from Gallaecio/constraints 
Fix master CI issues due to pip backtracking
 2021-03-19 10:34:36 +01:00
Adrián Chaves a390b934de Do not install mitmproxy in Python 3.9 2021-03-18 23:53:58 +01:00
Adrián Chaves 8e73e1dfb5 upper-constraints.txt: restrict botocore further 2021-03-18 23:42:29 +01:00
Adrián Chaves 94201612bc Simplify the get_retry_request code example 2021-03-18 23:35:47 +01:00
Adrián Chaves 1d83697990
Merge pull request #4769 from scrapy/http2 
Implement experimental HTTP/2 support
 2021-03-18 22:18:12 +01:00
vinayak 42e4dbb23d
Support Python 3.9 (#4759) 
* Update .travis.yml

* Update .travis.yml

* updage travis.yml

* Make 3.9 support official

* Upgrade mitmproxy for Python 3.9

* Restore the Pylint job

* Undo unintended change to mitmproxy requirement

* Enable Python 3.9 in GitHub Actions

* Work around reppy’s Python version limitation

* Disable tests in Windows / Python 3.9 due to a Twisted bug

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-03-18 09:40:03 -03:00
Eugenio Lacuesta 2f61d7cc03
Remove unnecesary del statement 2021-03-15 14:25:46 -03:00
Dmitriy Pomazunovskiy 9cc4513bd6 simpler stats access 2021-03-15 21:38:03 +06:00
Adrián Chaves c0f3ca1938 get_retry_request: add typing information 2021-03-12 14:05:44 +01:00
Dmitriy Pomazunovskiy 0f254a6afb Test from_settings 2021-03-12 17:11:50 +06:00
Dmitriy Pomazunovskiy d4b2b61255 Use from_settings for backward compatibility 2021-03-12 16:59:37 +06:00
Dmitriy Pomazunovskiy 6e5ea7924c Log skipped urls by length to INFO, add skipped stats 2021-03-12 11:08:41 +06:00
Mikhail Korobov ab037ab7c9
Merge pull request #5016 from deepang17/scrapy-4996 
DOCS: Cover scrapy-bench in the documentation
 2021-03-12 04:18:56 +05:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Adrián Chaves 3bea5e1a97 Remove unused _is_data_lost method 2021-03-09 16:19:51 +01:00
Adrián Chaves 5b2d3e17a3
Merge branch 'master' into http2 2021-03-09 14:59:17 +01:00
deepang17 2d41c77e6a Merge branch 'master' of https://github.com/scrapy/scrapy into scrapy-4996 2021-03-09 17:33:05 +05:30
deepang17 3d88ac605b FIX: Updated benchmarking.rst 2021-03-09 17:19:34 +05:30
Andrey Rahmatullin 954c4b48e6
Merge pull request #3667 from Gallaecio/long-allowed-domains 
Add a FAQ entry on how to deal with long lists of allowed domains
 2021-03-05 20:16:37 +05:00
Adrián Chaves dfd4ab2b6c
Merge pull request #3 from elacuesta/retry-request-customization 
Allow logger/stats customization in get_retry_request
 2021-03-03 16:30:08 +01:00
Andrey Rahmatullin d50bedbeea
Merge pull request #5022 from Gallaecio/fix-ci 
Limit tests to Twisted < 21
 2021-03-03 16:45:00 +05:00