mirror of https://github.com/scrapy/scrapy.git
Initial Python 2 removal (#4091)
This commit is contained in:
Andrey Rahmatullin
Adrián Chaves
parent
6d6da78eda
commit
229e722a03
10
.travis.yml
10
.travis.yml
|
|
@ -7,14 +7,6 @@ branches:
|
|||
- /^\d\.\d+\.\d+(rc\d+|\.dev\d+)?$/
|
||||
matrix:
|
||||
include:
|
||||
- env: TOXENV=py27
|
||||
python: 2.7
|
||||
- env: TOXENV=py27-pinned
|
||||
python: 2.7
|
||||
- env: TOXENV=py27-extra-deps
|
||||
python: 2.7
|
||||
- env: TOXENV=pypy
|
||||
python: 2.7
|
||||
- env: TOXENV=pypy3
|
||||
python: 3.5
|
||||
- env: TOXENV=py35
|
||||
|
|
@ -70,4 +62,4 @@ deploy:
|
|||
on:
|
||||
tags: true
|
||||
repo: scrapy/scrapy
|
||||
condition: "$TOXENV == py27 && $TRAVIS_TAG =~ ^[0-9]+[.][0-9]+[.][0-9]+(rc[0-9]+|[.]dev[0-9]+)?$"
|
||||
condition: "$TOXENV == py37 && $TRAVIS_TAG =~ ^[0-9]+[.][0-9]+[.][0-9]+(rc[0-9]+|[.]dev[0-9]+)?$"
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ https://scrapy.org
|
|||
Requirements
|
||||
============
|
||||
|
||||
* Python 2.7 or Python 3.5+
|
||||
* Python 3.5+
|
||||
* Works on Linux, Windows, Mac OSX, BSD
|
||||
|
||||
Install
|
||||
|
|
|
|||
|
|
@ -69,11 +69,11 @@ Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML pars
|
|||
What Python versions does Scrapy support?
|
||||
-----------------------------------------
|
||||
|
||||
Scrapy is supported under Python 2.7 and Python 3.5+
|
||||
Scrapy is supported under Python 3.5+
|
||||
under CPython (default Python implementation) and PyPy (starting with PyPy 5.9).
|
||||
Python 2.6 support was dropped starting at Scrapy 0.20.
|
||||
Python 3 support was added in Scrapy 1.1.
|
||||
PyPy support was added in Scrapy 1.4, PyPy3 support was added in Scrapy 1.5.
|
||||
Python 2 support was dropped in Scrapy 2.0.
|
||||
|
||||
.. note::
|
||||
For Python 3 support on Windows, it is recommended to use
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ Installation guide
|
|||
Installing Scrapy
|
||||
=================
|
||||
|
||||
Scrapy runs on Python 2.7 and Python 3.5 or above
|
||||
Scrapy runs on Python 3.5 or above
|
||||
under CPython (default Python implementation) and PyPy (starting with PyPy 5.9).
|
||||
|
||||
If you're using `Anaconda`_ or `Miniconda`_, you can install the package from
|
||||
|
|
@ -102,10 +102,8 @@ just like any other Python package.
|
|||
(See :ref:`platform-specific guides <intro-install-platform-notes>`
|
||||
below for non-Python dependencies that you may need to install beforehand).
|
||||
|
||||
Python virtualenvs can be created to use Python 2 by default, or Python 3 by default.
|
||||
|
||||
* If you want to install scrapy with Python 3, install scrapy within a Python 3 virtualenv.
|
||||
* And if you want to install scrapy with Python 2, install scrapy within a Python 2 virtualenv.
|
||||
Python virtualenvs can be created to use Python 2 by default, or Python 3 by default. As Scrapy
|
||||
only supports Python 3, make sure you created a Python 3 virtualenv.
|
||||
|
||||
.. _virtualenv: https://virtualenv.pypa.io
|
||||
.. _virtualenv installation instructions: https://virtualenv.pypa.io/en/stable/installation/
|
||||
|
|
@ -149,16 +147,12 @@ typically too old and slow to catch up with latest Scrapy.
|
|||
To install scrapy on Ubuntu (or Ubuntu-based) systems, you need to install
|
||||
these dependencies::
|
||||
|
||||
sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
|
||||
sudo apt-get install python3 python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
|
||||
|
||||
- ``python-dev``, ``zlib1g-dev``, ``libxml2-dev`` and ``libxslt1-dev``
|
||||
- ``python3-dev``, ``zlib1g-dev``, ``libxml2-dev`` and ``libxslt1-dev``
|
||||
are required for ``lxml``
|
||||
- ``libssl-dev`` and ``libffi-dev`` are required for ``cryptography``
|
||||
|
||||
If you want to install scrapy on Python 3, you’ll also need Python 3 development headers::
|
||||
|
||||
sudo apt-get install python3 python3-dev
|
||||
|
||||
Inside a :ref:`virtualenv <intro-using-virtualenv>`,
|
||||
you can install Scrapy with ``pip`` after that::
|
||||
|
||||
|
|
|
|||
|
|
@ -99,12 +99,12 @@ The storages backends supported out of the box are:
|
|||
|
||||
* :ref:`topics-feed-storage-fs`
|
||||
* :ref:`topics-feed-storage-ftp`
|
||||
* :ref:`topics-feed-storage-s3` (requires botocore_ or boto_)
|
||||
* :ref:`topics-feed-storage-s3` (requires botocore_)
|
||||
* :ref:`topics-feed-storage-stdout`
|
||||
|
||||
Some storage backends may be unavailable if the required external libraries are
|
||||
not available. For example, the S3 backend is only available if the botocore_
|
||||
or boto_ library is installed (Scrapy supports boto_ only on Python 2).
|
||||
library is installed.
|
||||
|
||||
|
||||
.. _topics-feed-uri-params:
|
||||
|
|
@ -182,7 +182,7 @@ The feeds are stored on `Amazon S3`_.
|
|||
* ``s3://mybucket/path/to/export.csv``
|
||||
* ``s3://aws_key:aws_secret@mybucket/path/to/export.csv``
|
||||
|
||||
* Required external libraries: `botocore`_ (Python 2 and Python 3) or `boto`_ (Python 2 only)
|
||||
* Required external libraries: `botocore`_
|
||||
|
||||
The AWS credentials can be passed as user/password in the URI, or they can be
|
||||
passed through the following settings:
|
||||
|
|
@ -399,6 +399,5 @@ format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
|
|||
|
||||
.. _URI: https://en.wikipedia.org/wiki/Uniform_Resource_Identifier
|
||||
.. _Amazon S3: https://aws.amazon.com/s3/
|
||||
.. _boto: https://github.com/boto/boto
|
||||
.. _botocore: https://github.com/boto/botocore
|
||||
.. _Canned ACL: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
|
||||
|
|
|
|||
|
|
@ -260,7 +260,7 @@ knowledge about Python internals. For more info about Guppy, refer to the
|
|||
|
||||
Debugging memory leaks with muppy
|
||||
=================================
|
||||
If you're using Python 3, you can use muppy from `Pympler`_.
|
||||
You can use muppy from `Pympler`_.
|
||||
|
||||
.. _Pympler: https://pypi.org/project/Pympler/
|
||||
|
||||
|
|
|
|||
|
|
@ -171,7 +171,7 @@ policy::
|
|||
|
||||
For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide.
|
||||
|
||||
Because Scrapy uses ``boto`` / ``botocore`` internally you can also use other S3-like storages. Storages like
|
||||
Because Scrapy uses ``botocore`` internally you can also use other S3-like storages. Storages like
|
||||
self-hosted `Minio`_ or `s3.scality`_. All you need to do is set endpoint option in you Scrapy settings::
|
||||
|
||||
AWS_ENDPOINT_URL = 'http://minio.example.com:9000'
|
||||
|
|
|
|||
|
|
@ -596,8 +596,8 @@ Response objects
|
|||
(for single valued headers) or lists (for multi-valued headers).
|
||||
:type headers: dict
|
||||
|
||||
:param body: the response body. To access the decoded text as str (unicode
|
||||
in Python 2) you can use ``response.text`` from an encoding-aware
|
||||
:param body: the response body. To access the decoded text as str you can use
|
||||
``response.text`` from an encoding-aware
|
||||
:ref:`Response subclass <topics-request-response-ref-response-subclasses>`,
|
||||
such as :class:`TextResponse`.
|
||||
:type body: bytes
|
||||
|
|
|
|||
|
|
@ -188,7 +188,6 @@ AWS_ENDPOINT_URL
|
|||
Default: ``None``
|
||||
|
||||
Endpoint URL used for S3-like storage, for example Minio or s3.scality.
|
||||
Only supported with ``botocore`` library.
|
||||
|
||||
.. setting:: AWS_USE_SSL
|
||||
|
||||
|
|
@ -199,7 +198,6 @@ Default: ``None``
|
|||
|
||||
Use this option if you want to disable SSL connection for communication with
|
||||
S3 or S3-like storage. By default SSL will be used.
|
||||
Only supported with ``botocore`` library.
|
||||
|
||||
.. setting:: AWS_VERIFY
|
||||
|
||||
|
|
@ -209,7 +207,7 @@ AWS_VERIFY
|
|||
Default: ``None``
|
||||
|
||||
Verify SSL connection between Scrapy and S3 or S3-like storage. By default
|
||||
SSL verification will occur. Only supported with ``botocore`` library.
|
||||
SSL verification will occur.
|
||||
|
||||
.. setting:: AWS_REGION_NAME
|
||||
|
||||
|
|
@ -219,7 +217,6 @@ AWS_REGION_NAME
|
|||
Default: ``None``
|
||||
|
||||
The name of the region associated with the AWS client.
|
||||
Only supported with ``botocore`` library.
|
||||
|
||||
.. setting:: BOT_NAME
|
||||
|
||||
|
|
|
|||
|
|
@ -72,8 +72,6 @@ scrapy.Spider
|
|||
spider that crawls ``mywebsite.com`` would often be called
|
||||
``mywebsite``.
|
||||
|
||||
.. note:: In Python 2 this must be ASCII only.
|
||||
|
||||
.. attribute:: allowed_domains
|
||||
|
||||
An optional list of strings containing domains that this spider is
|
||||
|
|
|
|||
|
|
@ -1,18 +0,0 @@
|
|||
parsel>=1.5.0
|
||||
PyDispatcher>=2.0.5
|
||||
w3lib>=1.17.0
|
||||
protego>=0.1.15
|
||||
|
||||
pyOpenSSL>=16.2.0 # Earlier versions fail with "AttributeError: module 'lib' has no attribute 'SSL_ST_INIT'"
|
||||
queuelib>=1.4.2 # Earlier versions fail with "AttributeError: '...QueueTest' object has no attribute 'qpath'"
|
||||
cryptography>=2.0 # Earlier versions would fail to install
|
||||
|
||||
# Reference versions taken from
|
||||
# https://packages.ubuntu.com/xenial/python/
|
||||
# https://packages.ubuntu.com/xenial/zope/
|
||||
cssselect>=0.9.1
|
||||
lxml>=3.5.0
|
||||
service_identity>=16.0.0
|
||||
six>=1.10.0
|
||||
Twisted>=16.0.0
|
||||
zope.interface>=4.1.3
|
||||
|
|
@ -14,8 +14,8 @@ del pkgutil
|
|||
|
||||
# Check minimum required Python version
|
||||
import sys
|
||||
if sys.version_info < (2, 7):
|
||||
print("Scrapy %s requires Python 2.7" % __version__)
|
||||
if sys.version_info < (3, 5):
|
||||
print("Scrapy %s requires Python 3.5" % __version__)
|
||||
sys.exit(1)
|
||||
|
||||
# Ignore noisy twisted deprecation warnings
|
||||
|
|
|
|||
7
setup.py
7
setup.py
|
|
@ -50,8 +50,6 @@ setup(
|
|||
'License :: OSI Approved :: BSD License',
|
||||
'Operating System :: OS Independent',
|
||||
'Programming Language :: Python',
|
||||
'Programming Language :: Python :: 2',
|
||||
'Programming Language :: Python :: 2.7',
|
||||
'Programming Language :: Python :: 3',
|
||||
'Programming Language :: Python :: 3.5',
|
||||
'Programming Language :: Python :: 3.6',
|
||||
|
|
@ -63,10 +61,9 @@ setup(
|
|||
'Topic :: Software Development :: Libraries :: Application Frameworks',
|
||||
'Topic :: Software Development :: Libraries :: Python Modules',
|
||||
],
|
||||
python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*',
|
||||
python_requires='>=3.5',
|
||||
install_requires=[
|
||||
'Twisted>=16.0.0;python_version=="2.7"',
|
||||
'Twisted>=17.9.0;python_version>="3.5"',
|
||||
'Twisted>=17.9.0',
|
||||
'cryptography>=2.0',
|
||||
'cssselect>=0.9.1',
|
||||
'lxml>=3.5.0',
|
||||
|
|
|
|||
|
|
@ -1,15 +0,0 @@
|
|||
# Tests requirements
|
||||
brotlipy
|
||||
jmespath
|
||||
mitmproxy==0.10.1
|
||||
mock
|
||||
netlib==0.10.1
|
||||
pytest
|
||||
pytest-cov
|
||||
pytest-twisted
|
||||
pytest-xdist
|
||||
testfixtures
|
||||
|
||||
# optional for shell wrapper tests
|
||||
bpython
|
||||
ipython<6.0
|
||||
53
tox.ini
53
tox.ini
|
|
@ -4,18 +4,16 @@
|
|||
# and then run "tox" from this directory.
|
||||
|
||||
[tox]
|
||||
envlist = py27
|
||||
envlist = py35
|
||||
|
||||
[testenv]
|
||||
deps =
|
||||
-ctests/constraints.txt
|
||||
-rrequirements-py2.txt
|
||||
-rrequirements-py3.txt
|
||||
-rtests/requirements-py3.txt
|
||||
# Extras
|
||||
botocore>=1.3.23
|
||||
google-cloud-storage
|
||||
leveldb
|
||||
Pillow>=3.4.2
|
||||
-rtests/requirements-py2.txt
|
||||
passenv =
|
||||
S3_TEST_FILE_URI
|
||||
AWS_ACCESS_KEY_ID
|
||||
|
|
@ -25,42 +23,8 @@ passenv =
|
|||
commands =
|
||||
py.test --cov=scrapy --cov-report= {posargs:scrapy tests}
|
||||
|
||||
[testenv:py27-pinned]
|
||||
basepython = python2.7
|
||||
deps =
|
||||
-ctests/constraints.txt
|
||||
cryptography==2.0
|
||||
cssselect==0.9.1
|
||||
lxml==3.5.0
|
||||
parsel==1.5.0
|
||||
Protego==0.1.15
|
||||
PyDispatcher==2.0.5
|
||||
pyOpenSSL==16.2.0
|
||||
queuelib==1.4.2
|
||||
service_identity==16.0.0
|
||||
six==1.10.0
|
||||
Twisted==16.0.0
|
||||
w3lib==1.17.0
|
||||
zope.interface==4.1.3
|
||||
-rtests/requirements-py2.txt
|
||||
# Extras
|
||||
botocore==1.3.23
|
||||
Pillow==3.4.2
|
||||
|
||||
[testenv:pypy]
|
||||
basepython = pypy
|
||||
commands =
|
||||
py.test {posargs:scrapy tests}
|
||||
|
||||
[testenv:py35]
|
||||
basepython = python3.5
|
||||
deps =
|
||||
-ctests/constraints.txt
|
||||
-rrequirements-py3.txt
|
||||
-rtests/requirements-py3.txt
|
||||
# Extras
|
||||
botocore>=1.3.23
|
||||
Pillow>=3.4.2
|
||||
|
||||
[testenv:py35-pinned]
|
||||
basepython = python3.5
|
||||
|
|
@ -86,19 +50,15 @@ deps =
|
|||
|
||||
[testenv:py36]
|
||||
basepython = python3.6
|
||||
deps = {[testenv:py35]deps}
|
||||
|
||||
[testenv:py37]
|
||||
basepython = python3.7
|
||||
deps = {[testenv:py35]deps}
|
||||
|
||||
[testenv:py38]
|
||||
basepython = python3.8
|
||||
deps = {[testenv:py35]deps}
|
||||
|
||||
[testenv:pypy3]
|
||||
basepython = pypy3
|
||||
deps = {[testenv:py35]deps}
|
||||
commands =
|
||||
py.test {posargs:scrapy tests}
|
||||
|
||||
|
|
@ -127,13 +87,6 @@ commands =
|
|||
|
||||
[testenv:py38-extra-deps]
|
||||
basepython = python3.8
|
||||
deps =
|
||||
{[testenv:py35]deps}
|
||||
reppy
|
||||
robotexclusionrulesparser
|
||||
|
||||
[testenv:py27-extra-deps]
|
||||
basepython = python2.7
|
||||
deps =
|
||||
{[testenv]deps}
|
||||
reppy
|
||||
|
|
|
|||
Loading…
Reference in New Issue