RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

320 Commits

Author SHA1 Message Date
Aditya 1c40dfa740 fix: handle CONNECTION_LOST & RESET separately 2020-07-07 15:36:41 +05:30
Aditya 7f5bb6b34c chore: add h2 to setup.py, tox.ini 
- Change log level for hpack to ERROR
 2020-07-06 13:12:04 +05:30
Adrián Chaves 3199048520 Complete Azure Pipelines CI setup 2020-07-03 00:56:24 +02:00
Aditya 6387445ef5 test(tox.ini): change Twisted -> Twisted[http2] 2020-06-28 18:44:57 +05:30
BroodingKangaroo 77333666f1 resolve conflicts 2020-06-26 14:55:46 +03:00
Eugenio Lacuesta b99fe4aa4c
Add google-cloud-storage to the 'pinned' tox environment 2020-06-19 21:41:15 -03:00
Eugenio Lacuesta 7babf359e0
Typing: Tox env, CI job 2020-06-18 13:56:07 -03:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
BroodingKangaroo 10ae1a284f Minor fixes 2020-05-15 22:50:54 +03:00
BroodingKangaroo 111a58fe3d Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries 2020-05-14 09:53:14 +03:00
Mikhail Korobov 794ab19660
Merge pull request #3727 from Gallaecio/pylint 
Use pylint
 2020-05-09 02:22:54 +05:00
Adrián Chaves a79cfce57b
Merge branch 'master' into pylint 2020-05-06 16:24:13 +02:00
Adrián Chaves 418b9b5f52 Travis CI: do not run security and Flake8 on multiple jobs 2020-05-06 11:15:02 +02:00
BroodingKangaroo df81870f39 Merge branch 'master' of https://github.com/scrapy/scrapy into ISSUE-4250-add_batch_deliveries 2020-05-01 12:18:39 +03:00
BroodingKangaroo 3f9874fac9 Add test s3 export 2020-05-01 11:52:16 +03:00
Adrián Chaves 5c0f11b4ef Simplify the asyncio Tox environment 2020-04-28 17:32:53 +02:00
Adrián Chaves 901892dab3 Fix the hoverxref configuration 2020-04-16 14:48:38 +02:00
Eugenio Lacuesta 11941c3244
Remove elusive six occurrence from tox.ini 2020-02-05 13:27:54 -03:00
Andrey Rakhmatullin 30ebd05a5f Simplify the tox asyncio entries. 2019-12-27 00:05:14 +05:00
Andrey Rakhmatullin 20289be810 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-20 19:31:20 +05:00
Adrián Chaves 00517ec273
Merge branch 'master' into docs-py38 2019-12-18 15:50:49 +01:00
Adrián Chaves 8c5a8a02ae
Merge branch 'master' into pylint 2019-12-18 15:43:40 +01:00
Mikhail Korobov 053319334c
Merge pull request #4179 from Gallaecio/user-friendlier-tox 
Make tox configuration more user friendly
 2019-12-18 18:35:19 +05:00
Vostretsov Nikita 12f9ffeb5d remove requirements-py3.txt 2019-12-18 10:53:27 +00:00
Andrey Rakhmatullin afc886e578 Simplify tox.ini asyncio entries. 2019-12-13 19:34:58 +05:00
Andrey Rakhmatullin c079d5002b Run tests without asyncio support by default, add py35-asyncio and py38-asyncio envs. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin cc19ab5439 Add tests that check asyncio support. 2019-12-04 21:32:56 +05:00
Adrián Chaves fc3af54dbd Make tox configuration more user friendly 2019-11-20 08:04:17 +01:00
Adrián Chaves 6d1667d5b8 Use the latest Python version to build the documentation 2019-11-18 11:13:34 +01:00
Mikhail Korobov 298d709e16
Merge pull request #4162 from Gallaecio/bandit 
Add bandit to CI
 2019-11-14 23:40:01 +05:00
Adrián Chaves 5ee5508cc3 Have CI record the 10 slowest tests 2019-11-14 15:42:34 +01:00
Adrián Chaves b8ef12cd47 Add bandit to CI 2019-11-14 12:11:28 +01:00
Adrián Chaves ecc46ac741
Merge branch 'master' into pylint 2019-11-11 10:45:25 +01:00
Adrián Chaves 1df5755699 Set the bases for testing code examples from the documentation 2019-11-08 16:00:10 +01:00
Adrián Chaves 40ed184914
Merge branch 'master' into pylint 2019-11-07 18:40:58 +01:00
Marc Hernández e8b1e46e85 Add pytest-flake8 (#3945) 2019-11-07 14:05:01 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
Roy 74909030a5 [tox.ini] Removing obsolete py37 extra deps enviornment 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:52:36 +00:00
Adrián Chaves 02577f55a0 Have PyLint cover all Python files in the repository 2019-10-24 13:25:11 +02:00
Adrián Chaves 3b6f7ac9f2 Use pylint 2019-10-22 19:44:11 +02:00
Roy 3d4317bfe4 [tox.ini] Added python 3.8 fields 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-21 18:32:30 +01:00
Vostretsov Nikita e418554c21 use proper equal 2019-09-09 08:12:32 +00:00
Vostretsov Nikita 7af8c76649 add pinned versions 2019-09-09 08:10:09 +00:00
Vostretsov Nikita 9578f49099 use protego as a default robots.txt parser 2019-09-09 07:36:55 +00:00
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Adrián Chaves c7b5ad0e20 Add a Sphinx extension to generate documentation coverage information 2019-06-04 15:03:25 +02:00
Daniel Graña 8ed6beb7f9 Needs to be installed within tox env 2018-12-20 19:39:29 -03:00
Konstantin Lopuhin 1d2e2735fc Merge branch 'master' of https://github.com/patiences/scrapy into patiences-master 2018-07-09 11:59:22 +03:00
Mikhail Korobov f11d65f7d6 TST make it clear which requirements are Python 2-only 
* rename requirements.txt to requirements-py2.txt, to make it clear they are Python 2-only
* make requirements-py3.txt consistent with requirements-py2.txt
 2018-06-29 18:34:11 +05:00
Mikhail Korobov 45f67eb64d TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow 2018-06-28 01:02:30 +05:00
Vostretsov Nikita c5ddfddb7e blacklist twisted version with regression in constraints file 2018-05-17 08:53:42 +00:00
Patience Shyu aca2655c12 [WIP] Run tests for Python 3.7 2018-03-02 14:57:39 +01:00
Konstantin Lopuhin f71df6f9ad Run tests for PyPy3 2017-12-25 14:48:06 +03:00
Hugo 44623687ab Drop support for EOL Python 3.3 2017-12-19 17:59:05 +02:00
Aditya 97d047a055 Fix link for Tox 2017-10-28 16:48:41 +05:30
rhoboro e5d4364b2a Add tests for GCS Storage 2017-09-13 16:24:16 +09:00
Paul Tremberth 1a18587d41 Jessi toxenv: Add cryptography as per https://packages.debian.org/jessie/python-cryptography 2017-07-24 19:30:08 +02:00
harshasrinivas 4ec07ae764 Create docs/requirements.txt 2017-03-20 22:21:08 +05:30
Mikhail Korobov de65ad3fb1 TST replace Ubuntu 12.04 tox environment with 14.04 2017-02-13 18:44:39 +05:00
Paul Tremberth 70f260d353 Don't run coverage stats when on PyPy 2017-02-01 15:19:17 +01:00
Paul Tremberth a214731471 Add Python 3.6 tox env + Travis CI build for it 2017-01-06 15:38:35 +01:00
Paul Tremberth 22e870e955 Add Debian Jessie test env 2016-09-01 10:19:49 +02:00
Paul Tremberth 1075587dbd Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 14:40:52 +02:00
Konstantin Lopuhin 32cd8c9165 add direct test for S3FilesStore 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 408bc1580b Pass env variables required for running tests against real s3 via tox. 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin c3fec83e7e use botocore by default, boto is still used in "precise" env 2016-02-15 19:59:47 +03:00
Paul Tremberth a93d49a64c Add Python 3.5 tox env and Python 3.4 tests in Travis CI 2016-01-13 13:09:16 +01:00
Mikhail Korobov 669be0a2b0 TST don't use broken Pillow version in tests 2015-10-06 17:24:52 +05:00
Daniel Graña cb9577a6d4 Do not be verbose with coverage report by default 2015-08-13 19:02:36 -03:00
Daniel Graña 108c3e63a5 Add coverage report trough codecov.io 2015-08-13 13:56:24 -03:00
Mikhail Korobov 75d3d03382 add service_identity to scrapy install_requires 2015-08-11 20:14:22 +05:00
nyov 4d41cc0dc4 PY3 split requirements into files 2015-08-08 04:19:37 +05:00
Julia Medina ec54942925 Create separate testenvs to build docs and check links 2015-05-14 20:54:26 -03:00
Julia Medina a555778013 Build docs in a temporary folder to avoid caching 2015-05-14 20:53:25 -03:00
Mikhail Korobov fbb1078f58 Merge pull request #1060 from Curita/python-logging 
[MRG+1] Python logging
 2015-04-29 23:20:34 +05:00
Daniel Graña 5eb098a939 Merge pull request #1168 from scrapy/service-identity 
install service_identity package in tests to prevent warnings
 2015-04-28 23:48:58 -03:00
Julia Medina c2d716807a Use LogCapture in testfixtures package for tests 
This allows to remove `get_testlog` helper, `flushLoggedErrors` from
twisted.trial.unittest.TestCase and Twisted log observers created for
each test on conftest.py.
 2015-04-22 17:24:40 -03:00
Julia Medina ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
Mikhail Korobov 271f7f54d5 TST install service_identity package in tests to prevent warnings 
Also, Twisted version is bumped for Python 3.x tests, just in case.

service_identity is not added to Scrapy requirements because Scrapy
supports older Twisted / PyOpenSSL versions which don't use it.
 2015-04-16 22:54:51 +05:00
Berker Peksag 31e5f164d4 Import unittest.mock if available. 
mock is in the stdlib since Python 3.3.
 2015-03-08 15:03:04 +02:00
Daniel Graña c05e99a4f4 oops, restore Pillow from precise test requirements 2014-09-10 12:21:08 -03:00
Daniel Graña a823207f18 Stop logobserver only when set 2014-09-10 12:09:07 -03:00
Daniel Graña 02dd4a56a2 fix requirement typo 2014-08-09 00:44:48 -03:00
Daniel Graña 37787081d8 precise ships zope.interface 3.6.1 2014-08-09 00:37:08 -03:00
Daniel Graña 3b64b2449e update other places where w3lib version is mentioned 2014-08-04 15:56:41 -03:00
Mikhail Korobov 51bd9f7d64 TST enable doctests for Python 3 2014-08-01 23:12:52 +06:00
Daniel Graña 7cd271eb8e remove more references to runtests.sh 2014-07-31 17:30:45 -03:00
Daniel Graña 70ee66ba0c Collect the list of packages not working on python3 2014-07-31 14:06:34 -03:00
Daniel Graña 99fb4eb12d fix requirements.txt for tests under precise 2014-07-31 12:50:07 -03:00
Daniel Graña aaf531c7dc fix Twisted package name 2014-07-30 19:45:08 -03:00
Daniel Graña eb1cf303d3 use pytest.ini to enable twisted plugin 2014-07-30 19:45:08 -03:00
Daniel Graña caab9f29f2 move tests requirements under tests dir 2014-07-30 19:45:08 -03:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00
Daniel Graña 0cf035dcbd Add tox env for python 3.4 2014-07-24 15:38:04 -03:00
Daniel Graña 79aba9e314 pytest 2.6.0 is on pypi now 2014-07-24 15:37:43 -03:00
Mikhail Korobov e38e3d2752 PY3 make it possible to run some tests in Python3 
* switch to Twisted 14.0.0 because SVN version broke twisted.log in Python 3;
* skip top-level shortcuts to make at least some tests pass;
* use PY3 compatible import of _monkeypatches.
 2014-07-02 22:23:01 +06:00
Daniel Graña 752787e634 Add a LevelDB cache backend 2014-06-04 16:09:07 -03:00
Daniel Graña 597fac25ec force installation of w3lib and queuelib for trunk env 2014-06-02 20:29:35 +00:00
Julia Medina c7c57178e0 New tox env: docs 2014-04-09 18:46:01 -03:00
Daniel Graña 986be9a396 Run testsuite with py.test 2014-04-03 12:26:44 -03:00
Mikhail Korobov 42dc34f924 TST Improved twisted installation in tox.ini for Python 3.3 
Install twisted trunk because released twisted doesn't install properly,
see https://twistedmatrix.com/trac/ticket/6539.

Also, test-requirements are removed for 3.3 because mitmproxy doesn't
install in Python 3.3, and mock is in Python 3.3 standard library.

Note that trial scrapy is not doing what is expected because twisted doesn't
have trial runner for 3.3 yet, so this command will most likely call "trial"
outside tox virtualenv and will use incorrect interpreter.
 2014-02-17 17:21:38 +06:00
Daniel Graña 02c80ca75b rename tests requirements to forward looking for /tests/ 2014-01-03 21:02:25 -02:00
Daniel Graña 958d0dbf39 move tests requirements to a single place 2014-01-03 18:55:54 -02:00
Mikhail Korobov b15df73a0f allow running individual tests via tox, e.g. «tox -e trunk -- scrapy.tests.test_spider» 2013-12-29 01:59:20 +06:00
duendex 247b330f08 Corrected typo in tox.ini 2013-12-02 21:34:35 -02:00
duendex 02bab270e8 Added mitmproxy as a requirement. 2013-12-02 20:25:12 -02:00
Daniel Graña 2564c21d4c add a tox env for Python 3.3 2013-11-19 20:15:15 -02:00
Daniel Graña fabb351097 map travis-ci matrix to tox environments 2013-11-07 10:58:08 -02:00
Daniel Graña 2df8156431 Drop Python 2.6 support 2013-10-29 13:44:00 -02:00
Daniel Graña 1f17230f17 add testing environment for development versions of w3lib and queuelib 2013-10-18 16:08:27 -02:00
Pablo Hoffman 892386ee01 tox.ini: disable sitepackages on windows, as a compiler is often not available 2013-08-12 18:59:41 -03:00
Mikhail Korobov 0d9f7843ba allow passing arguments to runtests script via tox 2013-07-09 00:41:13 +06:00
Mikhail Korobov f29e5d0ca1 reuse runtests.sh and runtests.bat scripts in tox.ini 2013-07-09 00:38:00 +06:00
Mikhail Korobov 3a6dff3090 make tox test running more consistent with bin/runtests.sh script 2013-07-09 00:30:39 +06:00
Mikhail Korobov b98e80cbb8 use tox deps instead of "manual" pip install; add lucid and precise environments to tox.ini 2013-07-09 00:26:59 +06:00
Mikhail Korobov 35fbec2e63 allow passing arguments to tox 
e.g. "tox -- scrapy.tests.test_contrib_loader" runs test_contrib_loader for Python 2.6 and 2.7
 2013-07-08 22:51:03 +06:00
Daniel Graña 5db45b3825 remove scrapyd, it was migrated to its own repository 2013-02-06 05:24:07 +00:00
Marc Abramowitz 9ae8ea96c4 Add tox.ini for tox (http://tox.testrun.org/) 2012-06-28 12:43:12 -07:00