RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,628 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

8628 Commits

Author SHA1 Message Date
vinayak 42e4dbb23d
Support Python 3.9 (#4759) 
* Update .travis.yml

* Update .travis.yml

* updage travis.yml

* Make 3.9 support official

* Upgrade mitmproxy for Python 3.9

* Restore the Pylint job

* Undo unintended change to mitmproxy requirement

* Enable Python 3.9 in GitHub Actions

* Work around reppy’s Python version limitation

* Disable tests in Windows / Python 3.9 due to a Twisted bug

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-03-18 09:40:03 -03:00
Mikhail Korobov ab037ab7c9
Merge pull request #5016 from deepang17/scrapy-4996 
DOCS: Cover scrapy-bench in the documentation
 2021-03-12 04:18:56 +05:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
deepang17 2d41c77e6a Merge branch 'master' of https://github.com/scrapy/scrapy into scrapy-4996 2021-03-09 17:33:05 +05:30
deepang17 3d88ac605b FIX: Updated benchmarking.rst 2021-03-09 17:19:34 +05:30
Andrey Rahmatullin 954c4b48e6
Merge pull request #3667 from Gallaecio/long-allowed-domains 
Add a FAQ entry on how to deal with long lists of allowed domains
 2021-03-05 20:16:37 +05:00
Andrey Rahmatullin d50bedbeea
Merge pull request #5022 from Gallaecio/fix-ci 
Limit tests to Twisted < 21
 2021-03-03 16:45:00 +05:00
Adrián Chaves 4fe26ae970 Limit tests to Twisted < 21 2021-03-02 19:46:34 +01:00
deepang17 36f1dbf665 DOCS: Covered scrapy-bench 2021-03-02 22:12:44 +05:30
deepang17 b25616d107 DOCS: Cover scrapy-bench in the documentation 2021-02-28 16:26:46 +05:30
deepang17 7a54580679 DOCS:Cover scrapy-bench in the documentation 2021-02-28 15:33:09 +05:30
Mikhail Korobov f95ebd8f7c
Merge pull request #5002 from djikoSal/refactor-curl-to-request-kwargs 
Refactor curl_to_request_kwargs #5001
 2021-02-25 02:00:55 +05:00
Mikhail Korobov bda5472943
Merge pull request #5008 from Gallaecio/close-file 
Close files in the PerYearXmlExportPipeline documentation example
 2021-02-25 01:49:59 +05:00
Adrián Chaves f689615e8d Close files in the PerYearXmlExportPipeline documentation example 2021-02-24 12:54:56 +01:00
Djiar 3894ebb149 Refactor curl_to_request_kwargs #5001 
Co-authored-by: alkazaz alkazaz@kth.se
Co-authored-by: swill swill@kth.se
Co-authored-by: lerjevik lerjevik@kth.se
Co-authored-by: aljica aljica@kth.se
 2021-02-24 11:04:48 +01:00
Wehzie a36f952198
fixed typo "an quotes.json" -> "a quotes.json" (#5005) 2021-02-24 08:15:44 +01:00
Andrey Rahmatullin fa3ebb1ced
Merge pull request #4987 from wRAR/asyncdef-tests 
Making sure we can run async def tests
 2021-02-18 18:34:25 +05:00
Andrey Rakhmatullin 49af7c4c8b Drop pytest-twisted, use Scrapy code to install the reactor. 2021-02-18 17:10:30 +05:00
Andrey Rakhmatullin abbbfbbb38 Add tests for deferred_f_from_coro_f. 2021-02-12 22:41:16 +05:00
Adrián Chaves ea92b49aaa
Merge pull request #4986 from wRAR/uvloop-py36 
Skip uvloop 0.15.0+ on py36.
 2021-02-11 12:07:28 +01:00
Andrey Rakhmatullin 54fd371481 Skip uvloop 0.15.0+ on py36. 2021-02-11 14:24:11 +05:00
Adrián Chaves de82ca8547
Merge pull request #4982 from wRAR/refactor-spidermiddlewaremanager 
Refactor SpiderMiddlewareManager.scrape_response.
 2021-02-08 21:27:21 +01:00
Andrey Rakhmatullin 1e9b52c3e0 Refactor SpiderMiddlewareManager.scrape_response. 2021-02-08 22:02:03 +05:00
Mikhail Korobov 904a50138b
Merge pull request #4973 from Gallaecio/zyte 
Scrapinghub → Zyte
 2021-02-03 01:10:52 +05:00
Adrián Chaves f30f53b3cc Scrapinghub → Zyte 2021-02-02 15:03:20 +01:00
Mikhail Korobov 28262d4b24
Merge pull request #4956 from Gallaecio/readme-contributors 
Mention contributors in the README
 2021-01-14 19:58:52 +05:00
Adrián Chaves 6e7ae789f9
Reuse the text from https://scrapy.org/ 2021-01-14 11:59:38 +01:00
M Ikram Ullah Khan 0a1e2fefab
Docs: fix typo in news.rst (#4942) 2021-01-04 10:30:23 -03:00
Eugenio Lacuesta 80db569aea
Migrate CI to GitHub actions (#4924) 2021-01-01 19:13:39 -03:00
Kader DJEHAF 44a7ab5bf0
Fix warning: Expected type 'bool', got 'int' instead (#4940) 
* Fix warning: Expected type 'bool', got 'int' instead

* Update defer.py
 2020-12-30 11:22:27 -03:00
Adrián Chaves 39bc9a4d28
Merge pull request #4814 from dswij/tox-pip20.2 
add pip 20.2 test for tox
 2020-12-24 15:31:42 +01:00
Adrián Chaves 24e1b350ae
Merge pull request #4936 from timgates42/bugfix_typo_without 
docs: fix simple typo, wihout -> without
 2020-12-22 13:57:03 +01:00
Tim Gates 6dccf82eaa
docs: fix simple typo, wihout -> without 
There is a small typo in scrapy/http/request/form.py.

Should read `without` rather than `wihout`.
 2020-12-22 07:49:13 +11:00
Eugenio Lacuesta b83a1a6fbf Disable test under pypy 2020-12-21 17:21:08 +01:00
Adrián Chaves 45eb099ed1 Maybe it’s about having a newer libssl 2020-12-21 15:37:02 +01:00
Adrián Chaves 1c1255a75d Use sudo for apt-get 2020-12-21 14:41:02 +01:00
Adrián Chaves 798a818caf Move apt-get command from Tox to Travis CI 2020-12-21 13:35:40 +01:00
Adrián Chaves 0dff5781bc Blind attempt to fix the build of the cryptography-provided OpenSSL 2020-12-21 11:13:14 +01:00
Adrián Chaves 0567fdc56e Merge remote-tracking branch 'upstream/master' into tox-pip20.2 2020-12-21 11:02:05 +01:00
Adrián Chaves d0af008608
Merge pull request #4878 from elacuesta/response-protocol-attribute 
Response.protocol attribute
 2020-12-16 15:27:40 +01:00
Mikhail Korobov 212163e199
Merge pull request #4912 from elacuesta/remove_deprecated_pickled_settings_to_override 
Remove deprecated SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE
 2020-12-04 01:36:47 +05:00
gunadhya db10aaf9eb
Update links in installation guide (#4899) 2020-12-03 10:56:36 +01:00
Eugenio Lacuesta 6091f3cc03
Remove unused pickle import 2020-12-01 10:26:21 -03:00
Eugenio Lacuesta a80bafe5cd
Remove deprecated SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE 2020-11-30 19:03:13 -03:00
Kader DJEHAF 7fec9f991f
[Cleaned] PEP 8: E251 unexpected spaces around keyword / parameter equals (#4911) 
[Cleaned] PEP 8: E251 unexpected spaces around keyword / parameter equals
 2020-11-30 17:47:28 -03:00
Adrián Chaves 487e19528b
Merge pull request #4901 from elacuesta/remove_deprecated_stuff 
Remove deprecated stuff
 2020-11-30 14:41:40 +01:00
Mikhail Korobov aed1707f24
Merge pull request #4909 from etimoz/exporters-doc-super-fix 
Removed wrong super argument in overriding serialize_fields code example
 2020-11-29 23:09:20 +05:00
etimoz 95d39d5cb4 removed wrong super argument in overriding serialize_fields code example 2020-11-29 13:24:04 +01:00
Andrey Rahmatullin 440e45d172
Merge pull request #4900 from elacuesta/deprecate_utils_py36 
Deprecate scrapy.utils.py36 module
 2020-11-24 23:59:00 +05:00
Adrián Chaves f6879c681e SCRAPY_PINNED → _SCRAPY_PINNED 2020-11-23 23:53:03 +01:00