RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,251 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

95 Commits

Author SHA1 Message Date
Aditya 7f5bb6b34c chore: add h2 to setup.py, tox.ini 
- Change log level for hpack to ERROR
 2020-07-06 13:12:04 +05:30
Aditya 7b1ad995a4 test: query params, certificate & ip_address 
- refactor from str.format() to f-strings
 2020-07-01 10:45:36 +05:30
Aditya 23906b6bee refactor: move TypedDict types to types.py 
- rename LOGGER -> logger
- remove self._write_to_transport from Stream class and handle all
transport related activities inside HTTP2ClientProtocol class
 2020-06-29 18:21:05 +05:30
Aditya 690dd7f38b test: GET & POST request test for h2 client 
- Remove repeated dependency Twisted from setup.py
- Test for both GET & POST when
  - Only 1 request
  - Large number (=20) of requests
and
  - Small Data (10 KB) per request
  - Large Data (10 MB) per request
- Test when request is cancelled by the client'

BREAKING CHANGES
Tests raises OpenSSL.SSL.Error when run using tox. However, all tests
passes when ran using `python -m unittest`.
 2020-06-28 16:38:49 +05:30
Aditya 69f6d038c0 feat: TypedDict for Stream._response 
- remove test_protocol.py as working testing environment is setup 🙂🙃
- Add typing_extensions as dependency to support TypedDict for
python<3.8
 2020-06-24 07:50:21 +05:30
Aditya 700df3eeb7 test: mockserver with h2 protocol for tests 
- add Twisted[http2] in setup.py requirements
- add test_protocol.py to test the current implementation

BREAKING CHANGES
test_download times out because of no protocol negotiated between
Mockserver and HTTP/2 client
 2020-06-17 21:02:14 +05:30
Vostretsov Nikita 16f2cb4a83 project URLs in machine-readable format for showing in pypi 2020-03-26 12:57:39 +00:00
Eugenio Lacuesta 0c4e5b68ea
Remove six from requirements and setup files 2019-11-03 12:30:34 -03:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
Roy 9b47dc6a70 [travis, setup] Adding official python 3.8 support 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 11:24:52 +00:00
Roy Healy 179dc916ff
Update setup.py 
Updating setup.py to remove python 3.8 support for now
 2019-10-27 16:48:20 +00:00
Roy 4e939ca75d [setup.py] Added python 3.8 fields 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-21 18:33:18 +01:00
Vostretsov Nikita c7f2bdfdbe add protego to install_requires 2019-09-10 08:58:52 +00:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Renne Rocha a25e09ecdd Added constrain on lxml version based on Python version 2019-07-30 23:24:41 -03:00
Adrián Chaves 0da972339b Require Twisted<=19.2.0 for Python 3.4 2019-06-11 14:11:38 +02:00
Mikhail Korobov cdd04dfb1d declare Python 3.7 support in setup.py 2018-12-26 13:13:49 +05:00
Mikhail Korobov 25ac4691b4 require parsel 1.5+ 2018-09-12 17:57:27 +05:00
Eugenio Lacuesta 0d87e77afe Bump parsel dependency 2018-02-08 15:03:57 -03:00
Mikhail Korobov 1058169f0e setup.py: mention that we support PyPy. See GH-2213. 2017-12-25 15:31:07 +05:00
Hugo 44623687ab Drop support for EOL Python 3.3 2017-12-19 17:59:05 +02:00
Aditya 9d9d83a8c3 Use https links wherever possible 2017-10-28 16:24:40 +05:30
Konstantin Lopuhin b0a9236357 Use environment markers for custom PyPy requirements 2017-06-19 19:16:50 +03:00
Mikhail Korobov e1ceaf3b5f require w3lib 1.17+ 2017-02-15 00:32:44 +05:00
Paul Tremberth 1295c17a26 Bump parsel requirement to at least parsel v1.1 2017-02-02 17:47:22 +01:00
Paul Tremberth b3406677b9 Update classifiers in setup.py 2017-01-09 14:40:02 +01:00
Paul Tremberth 78d34cc44c Merge pull request #2454 from crasker/fix/low_parsel_version_number 
[MRG+1] Fix/low parsel version number
 2016-12-19 17:11:17 +01:00
zhouyc 2b3abdb700 update parsel version which will cause attribute error 2016-12-19 16:42:31 +08:00
nyov c8cf1a303d Bump Twisted dependency to 13.1.0 (released June 2013) 2016-12-01 22:02:11 +00:00
Ashish Kulkarni bb3b806467 Use w3lib.url.canonicalize_url() from w3lib 1.15.0 
Also remove code/imports which are now unused due to this change.

fixes #2157
 2016-08-16 17:42:16 +05:30
Paul Tremberth c8bff23751 Bump w3lib version dependency in setup.py 2016-04-29 10:29:37 +02:00
nyov 3787fec460 Remove duplicate code now handled by newer w3lib 
see f3029a6a10
 2016-03-30 14:58:03 +00:00
Mikhail Korobov 17d3bec699 declare Python 3 support in setup.py 2016-03-01 16:34:13 +05:00
Elias Dorneles 26ebccd37a upgrade parsel and use its function to instantiate root for finding form 2015-08-11 14:09:34 -03:00
Elias Dorneles 2fe6d128f5 upgrade parsel and using promoted root attribute 2015-08-11 14:09:16 -03:00
Elias Dorneles 17d7347a36 update minimal parsel version, add deprecated classes for csstranslator modules, fix test 2015-08-11 14:09:00 -03:00
Elias Dorneles ce21884a97 migrating scrapy Selector to use Parsel 2015-08-11 14:08:43 -03:00
Mikhail Korobov 75d3d03382 add service_identity to scrapy install_requires 2015-08-11 20:14:22 +05:00
Daniel Graña 776064aee1 PY3 port signals 2015-07-25 18:04:55 +02:00
Elias Dorneles 3d3633f3d2 favoring web scraping over screen scraping in the descriptions 2015-04-25 11:20:20 -03:00
Pablo Hoffman 5846d6154c emphasize web crawling over screen scraping on scrapy description. closes #586 2015-03-18 00:24:16 -03:00
Daniel Graña 0254f58763 Merge branch 'moderm-setuppy' 2014-08-08 18:09:43 -03:00
Daniel Graña 029c51acef There is a trove classifier for Scrapy framework! 
Added by https://bitbucket.org/pypa/pypi/issue/179
 2014-08-05 14:43:13 -03:00
Paul Tremberth 480cfa1994 Update w3lib requirement to 1.8.0 2014-08-04 16:01:28 +02:00
Daniel Graña fcd34b6561 set zip_safe=False 2014-08-01 16:26:08 -03:00
Daniel Graña 1fc4e59cf4 do not ship tests package 2014-08-01 16:23:37 -03:00
Daniel Graña a8f45dc6dd Modernize setup.py 2014-08-01 15:43:33 -03:00
Daniel Graña bfe02b019e Revert "limit Twisted support to pre-14.0.0 while #718 is fixed" 
This reverts commit 65f69e1609.
 2014-06-25 04:22:18 -03:00
Daniel Graña 65f69e1609 limit Twisted support to pre-14.0.0 while #718 is fixed 2014-06-02 13:27:20 -03:00
alexanderlukanin13 ab6496647c Added "six>=1.5.2" to requirements 2014-01-15 13:05:00 +06:00