RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

9291 Commits

Author SHA1 Message Date
BroodingKangaroo f1020e0e6a Tiny changes 2020-07-06 15:40:53 +03:00
Adrián Chaves 17aec5944c
Update tests/CrawlerRunner/ip_address.py 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2020-07-06 10:47:25 +02:00
Adrián Chaves ec06cf79a6
Update tests/CrawlerRunner/ip_address.py 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2020-07-06 10:47:11 +02:00
Aditya 7f5bb6b34c chore: add h2 to setup.py, tox.ini 
- Change log level for hpack to ERROR
 2020-07-06 13:12:04 +05:30
Aditya a94b30342a test: reduce test data size to 1MB 2020-07-06 12:49:12 +05:30
BroodingKangaroo 6454d456d2 Make check of placeholder less strict 2020-07-03 10:24:13 +03:00
Adrián Chaves eb93774256 TrackrefTestCase.test_get_oldest: protect from lack of precision 2020-07-03 01:41:47 +02:00
Adrián Chaves 3199048520 Complete Azure Pipelines CI setup 2020-07-03 00:56:24 +02:00
Adrián Chaves 6e58da1dcd
Merge branch 'master' into azure-pipelines 2020-07-02 17:49:42 +02:00
BroodingKangaroo 1e245046ed Change setting name. Add leading zeroes to batch_id. Minor fixes. 2020-07-02 12:38:08 +03:00
Mikhail Korobov 56a6d22352
Merge pull request #4653 from scrapy/openssl-man-update 
Update the OpenSSL cipher list format link
 2020-07-01 23:28:52 +05:00
Andrey Rahmatullin af55d23167
Update the OpenSSL cipher list format link 
OpenSSL `ciphers(1)` is now almost empty: https://www.openssl.org/docs/manmaster/man1/ciphers.html

Alternative would be linking to 1.1.1 docs specifically.
 2020-07-01 19:46:54 +05:00
Aditya 4acdc2e5d6 refactor: use __qualname__, () for large strings 2020-07-01 20:15:33 +05:30
Aditya c361fe0d3b feat: check for invalid hostname 
- Initiating requests having hostname or (ip_address, port) different
from the peer to which HTTP/2 connection is made can lead to closing the
whole connection and close out all the pending streams.
- This change aims to fix that problem
- Add required tests
- Save hostname & port in H2ConnectionMetadataDict
 2020-07-01 18:14:44 +05:30
Andrey Rahmatullin 0379df8608
Merge pull request #4652 from elacuesta/docs-loaders-dataclass 
Docs: Simplify dataclass example in item loader page
 2020-07-01 17:35:33 +05:00
ajaymittur28 065b9b1170 Update regex import 2020-07-01 15:53:29 +05:30
BroodingKangaroo 7b1d3c35ea Minor updates 2020-07-01 12:24:31 +03:00
ajaymittur28 006a945214 Update schemaless http proxy test 2020-07-01 13:32:58 +05:30
ajaymittur28 7fc80671a8 Update schemaless URI support 2020-07-01 13:32:17 +05:30
Aditya 7b1ad995a4 test: query params, certificate & ip_address 
- refactor from str.format() to f-strings
 2020-07-01 10:45:36 +05:30
Eugenio Lacuesta 5b88c522ac
Simplify dataclass example in item loader docs 2020-06-30 12:18:21 -03:00
Aditya 50dd9271b4 fix: disable redundant logs 
- while testing the job exceeded the maximum log length
and was terminated
- reduce the number of requests from 20 to 10
 2020-06-30 07:36:56 +05:30
Aditya 26ab3e4137 feat: FIFO policy to handle large no. of requests 
- add required test -- test by sending 1000 requests
- increase test timeout to 180 seconds to account for tests taking long
time
 2020-06-30 06:44:20 +05:30
Aditya d17417b03a Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-06-29 18:43:42 +05:30
Aditya Kumar e46b47c365
Renew the localhost certificate for tests (#4650) 
Validity
  Not Before: Jun 28 12:54:15 2020 GMT
  Not After : Jun 28 12:54:15 2021 GMT
Subject: C = IE, O = Scrapy, CN = localhost
 2020-06-29 15:05:13 +02:00
Aditya 90a7007f88 test: warnsize logs, no content header, dataloss 2020-06-29 18:29:31 +05:30
Aditya 23906b6bee refactor: move TypedDict types to types.py 
- rename LOGGER -> logger
- remove self._write_to_transport from Stream class and handle all
transport related activities inside HTTP2ClientProtocol class
 2020-06-29 18:21:05 +05:30
Marc Hernández 464f24f8c1
Add --data-raw to utils.curl and fix missing method with data (#4612) 2020-06-29 14:20:29 +02:00
Aditya 6387445ef5 test(tox.ini): change Twisted -> Twisted[http2] 2020-06-28 18:44:57 +05:30
Aditya 690dd7f38b test: GET & POST request test for h2 client 
- Remove repeated dependency Twisted from setup.py
- Test for both GET & POST when
  - Only 1 request
  - Large number (=20) of requests
and
  - Small Data (10 KB) per request
  - Large Data (10 MB) per request
- Test when request is cancelled by the client'

BREAKING CHANGES
Tests raises OpenSSL.SSL.Error when run using tox. However, all tests
passes when ran using `python -m unittest`.
 2020-06-28 16:38:49 +05:30
BroodingKangaroo 05c2587c6a Docs update and tiny fixes 2020-06-28 09:45:45 +03:00
ajaymittur28 f53f06020b Test http schemaless proxy 2020-06-27 23:28:40 +05:30
ajaymittur28 23da8e1068 Add schemaless http proxy support 2020-06-27 20:36:45 +05:30
BroodingKangaroo 88a52198b9 Add batch_item_count support in FEEDS setting 2020-06-27 12:45:08 +03:00
BroodingKangaroo c3cee74fd4 Change default value of FEED_STORAGE_BATCH_ITEM_COUNT to 0 2020-06-26 18:45:21 +03:00
BroodingKangaroo 77333666f1 resolve conflicts 2020-06-26 14:55:46 +03:00
Mikhail Korobov 0c8d8c5e85
Merge pull request #4645 from Lukas0907/fix-starturlsspider 
Spider constructor expects name as argument, not start_urls.
 2020-06-25 18:36:43 +05:00
Adrián Chaves 9f60481360 Bump version: 2.1.0 → 2.2.0 2020-06-24 12:27:39 +02:00
Adrián Chaves 6f4c964aa4
Cover Scrapy 2.2.0 in the release notes (#4630) 2020-06-24 12:26:38 +02:00
Aditya 065b3153fe Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-06-24 07:51:41 +05:30
Aditya 69f6d038c0 feat: TypedDict for Stream._response 
- remove test_protocol.py as working testing environment is setup 🙂🙃
- Add typing_extensions as dependency to support TypedDict for
python<3.8
 2020-06-24 07:50:21 +05:30
Aditya a97ac0adf8 test: GET request for HTTP2Client using mockserver 2020-06-24 06:40:20 +05:30
Lukas Anzinger 3672f5f988 Spider constructor expects name as argument, not start_urls. 
Fixes #4644
 2020-06-23 14:56:12 +02:00
Andrey Rahmatullin 536643efde
Merge pull request #4629 from StasDeep/fix/duplicated-feed-logs 
Fix duplicated feed logs
 2020-06-23 15:33:48 +05:00
Andrey Rahmatullin 4bee73cfa7
Merge pull request #4637 from elacuesta/typing 
Typing: Tox env, CI job
 2020-06-23 15:31:37 +05:00
Andrey Rahmatullin 374d996093
Merge pull request #4642 from elacuesta/docs-loaders-dataclass 
Docs: add note about dataclass items and loaders
 2020-06-23 15:30:48 +05:00
Adrián Chaves cfd039aeb6 Remove a duplicate GCS_PROJECT_ID reference target 2020-06-22 19:28:33 +02:00
Eugenio Lacuesta 73b6ce8cb5
Update docs about dataclass items and loaders 2020-06-22 14:13:37 -03:00
Eugenio Lacuesta 1335d9053e
Update docs/topics/loaders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-06-22 14:05:44 -03:00
Eugenio Lacuesta 3efea98e05
Docs: add note about dataclass items and loaders 2020-06-22 12:41:25 -03:00