RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,814 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

8814 Commits

Author SHA1 Message Date
Adrián Chaves 8e73e1dfb5 upper-constraints.txt: restrict botocore further 2021-03-18 23:42:29 +01:00
Adrián Chaves 94201612bc Simplify the get_retry_request code example 2021-03-18 23:35:47 +01:00
Adrián Chaves 1d83697990
Merge pull request #4769 from scrapy/http2 
Implement experimental HTTP/2 support
 2021-03-18 22:18:12 +01:00
vinayak 42e4dbb23d
Support Python 3.9 (#4759) 
* Update .travis.yml

* Update .travis.yml

* updage travis.yml

* Make 3.9 support official

* Upgrade mitmproxy for Python 3.9

* Restore the Pylint job

* Undo unintended change to mitmproxy requirement

* Enable Python 3.9 in GitHub Actions

* Work around reppy’s Python version limitation

* Disable tests in Windows / Python 3.9 due to a Twisted bug

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-03-18 09:40:03 -03:00
Eugenio Lacuesta 2f61d7cc03
Remove unnecesary del statement 2021-03-15 14:25:46 -03:00
Dmitriy Pomazunovskiy 9cc4513bd6 simpler stats access 2021-03-15 21:38:03 +06:00
Adrián Chaves c0f3ca1938 get_retry_request: add typing information 2021-03-12 14:05:44 +01:00
Dmitriy Pomazunovskiy 0f254a6afb Test from_settings 2021-03-12 17:11:50 +06:00
Dmitriy Pomazunovskiy d4b2b61255 Use from_settings for backward compatibility 2021-03-12 16:59:37 +06:00
Dmitriy Pomazunovskiy 6e5ea7924c Log skipped urls by length to INFO, add skipped stats 2021-03-12 11:08:41 +06:00
Mikhail Korobov ab037ab7c9
Merge pull request #5016 from deepang17/scrapy-4996 
DOCS: Cover scrapy-bench in the documentation
 2021-03-12 04:18:56 +05:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Adrián Chaves 3bea5e1a97 Remove unused _is_data_lost method 2021-03-09 16:19:51 +01:00
Adrián Chaves 5b2d3e17a3
Merge branch 'master' into http2 2021-03-09 14:59:17 +01:00
deepang17 2d41c77e6a Merge branch 'master' of https://github.com/scrapy/scrapy into scrapy-4996 2021-03-09 17:33:05 +05:30
deepang17 3d88ac605b FIX: Updated benchmarking.rst 2021-03-09 17:19:34 +05:30
Andrey Rahmatullin 954c4b48e6
Merge pull request #3667 from Gallaecio/long-allowed-domains 
Add a FAQ entry on how to deal with long lists of allowed domains
 2021-03-05 20:16:37 +05:00
Adrián Chaves dfd4ab2b6c
Merge pull request #3 from elacuesta/retry-request-customization 
Allow logger/stats customization in get_retry_request
 2021-03-03 16:30:08 +01:00
Andrey Rahmatullin d50bedbeea
Merge pull request #5022 from Gallaecio/fix-ci 
Limit tests to Twisted < 21
 2021-03-03 16:45:00 +05:00
Adrián Chaves 4fe26ae970 Limit tests to Twisted < 21 2021-03-02 19:46:34 +01:00
deepang17 36f1dbf665 DOCS: Covered scrapy-bench 2021-03-02 22:12:44 +05:30
Eugenio Lacuesta 9e62355271
Allow logger/stats customization in get_retry_request 2021-03-02 12:12:22 -03:00
James McKinney 3c5668d0db
docs: Clarify there's one extension instance per spider 2021-03-01 22:00:33 -05:00
deepang17 b25616d107 DOCS: Cover scrapy-bench in the documentation 2021-02-28 16:26:46 +05:30
deepang17 7a54580679 DOCS:Cover scrapy-bench in the documentation 2021-02-28 15:33:09 +05:30
Mikhail Korobov f95ebd8f7c
Merge pull request #5002 from djikoSal/refactor-curl-to-request-kwargs 
Refactor curl_to_request_kwargs #5001
 2021-02-25 02:00:55 +05:00
Mikhail Korobov bda5472943
Merge pull request #5008 from Gallaecio/close-file 
Close files in the PerYearXmlExportPipeline documentation example
 2021-02-25 01:49:59 +05:00
Adrián Chaves f689615e8d Close files in the PerYearXmlExportPipeline documentation example 2021-02-24 12:54:56 +01:00
Djiar 3894ebb149 Refactor curl_to_request_kwargs #5001 
Co-authored-by: alkazaz alkazaz@kth.se
Co-authored-by: swill swill@kth.se
Co-authored-by: lerjevik lerjevik@kth.se
Co-authored-by: aljica aljica@kth.se
 2021-02-24 11:04:48 +01:00
Adrián Chaves 386e2a51ae tests/test_downloader_handlers_http2.py: fix style issue 2021-02-24 10:41:01 +01:00
Adrián Chaves 12064d799b HTTP/2: improve header handling 2021-02-24 10:37:38 +01:00
Wehzie a36f952198
fixed typo "an quotes.json" -> "a quotes.json" (#5005) 2021-02-24 08:15:44 +01:00
Adrián Chaves 5101094207 HTTP/2: test a CONNECT request 2021-02-24 07:33:39 +01:00
Adrián Chaves 5ba31cd1a2 HTTP/2 stream close reason handling: Use else + assert instead of elif 2021-02-23 11:57:33 +01:00
Adrián Chaves bd29f32dee HTTP/2: do not make conn_lost_deferred optional 2021-02-23 06:42:28 +01:00
Adrián Chaves 7605f19ec4 HTTP/2: test 2 concurrent requests to the same domain 2021-02-23 05:54:48 +01:00
Eugenio Lacuesta 6326178bc5
http2: acceptable protocol update, tests (#4994) 2021-02-22 16:50:51 +01:00
Adrián Chaves 1f7665c4cf Silence a PyLint check on a mistake made for testing purposes 2021-02-22 16:48:10 +01:00
Adrián Chaves 722a33a2ac Fix style issues 2021-02-22 16:42:38 +01:00
Adrián Chaves 80f5003c88 Add tests for get_retry_request 2021-02-22 16:38:38 +01:00
Adrián Chaves 6ab990181c Document get_retry_requests 2021-02-22 14:48:03 +01:00
Adrián Chaves ec836dcc92 Solve style issues 2021-02-22 14:15:28 +01:00
Adrián Chaves 3f5a1956d8 Merge remote-tracking branch 'upstream/master' into retry-request 2021-02-22 14:10:26 +01:00
Adrián Chaves 825462615a get_retry_request: set the default retry reason to “unspecified” 2021-02-22 14:09:48 +01:00
Adrián Chaves 5fc27b1e6f Remove RetrySpiderMixin and retry_request 2021-02-22 14:09:06 +01:00
Andrey Rahmatullin fa3ebb1ced
Merge pull request #4987 from wRAR/asyncdef-tests 
Making sure we can run async def tests
 2021-02-18 18:34:25 +05:00
Andrey Rakhmatullin 49af7c4c8b Drop pytest-twisted, use Scrapy code to install the reactor. 2021-02-18 17:10:30 +05:00
Eugenio Lacuesta 4418f78941
Simplify check for negotiated protocol 
negotiatedProtocol's type is Optional[bytes]

See https://github.com/twisted/twisted/blob/twisted-20.3.0/src/twisted/protocols/tls.py#L563-L587
and https://www.pyopenssl.org/en/20.0.1/api/ssl.html#OpenSSL.SSL.Connection.get_alpn_proto_negotiated

Note that OpenSSL.SSL.Connection.get_next_proto_negotiated is deprecated:
https://www.pyopenssl.org/en/20.0.0/changelog.html#backward-incompatible-changes
 2021-02-17 18:36:52 -03:00
Eugenio Lacuesta e80f37bd3f
Test http2 agent for unsupported scheme 2021-02-17 16:34:29 -03:00
Eugenio Lacuesta ac82a4a8a0
Merge remote-tracking branch 'upstream/master' into http2 2021-02-17 16:33:57 -03:00