RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,251 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

100 Commits

Author SHA1 Message Date
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta 8b549392f9
Bump minimum Python version to 3.5.2 (#4615) 2020-06-11 14:53:59 +02:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Mikhail Korobov 14612fc3d3
Merge pull request #4534 from elacuesta/deprecate-baseitem 
Deprecate scrapy.item.BaseItem
 2020-05-16 03:24:06 +05:00
Adrián Chaves 36c3c9713e
Run tests with Python 3.5.0 (#4518) 
* Run tests with Python 3.5.0

* Use Ubuntu Trusty 14.04 to test Python 3.5.0

* Use mitmproxy<3.0.0 on Python 3.5.0

* Skip tests requiring mitmproxy in Python 3.5.0

* Change the minimum Python version from 3.5 to 3.5.1

* Do not prevent Scrapy from working with Python 3.5.0

* Force system Python 3.5.1

* Do not install a non-system Python in Python 3.5.1 jobs

* Switch to Trusty to be able to test Xenial’s Python version ¯\_(ツ)_/¯

* Add missing trusty

* Stop breaking old PyPy

* Allow installing Scrapy on Python 3.5.0
 2020-05-15 22:37:56 +05:00
Eugenio Lacuesta 7988c676a9
Update Item docstring, update BaseItem occurrences 2020-05-05 13:11:01 -03:00
nyov a34c366fa4 DOC linkcheck run; https and 301 link updates. 
Closes #4359
 2020-02-25 16:51:33 +00:00
elacuesta 35dafef7f1
Specify Twisted reactor (TWISTED_REACTOR setting) (#4294) 
* Add the ability to install a specific reactor

* Add docs for the TWISTED_REACTOR setting

* Add tests for the TWISTED_REACTOR setting

* Update asyncio reactor test

* Ignore W503 globally

W503 is not PEP8-compliant:
c59c4376ad

* Line length adjustment

* Adjust asyncio reactor tests

* Merge ASYNCIO_ENABLED and TWISTED_REACTOR settings

* More docs about TWISTED_REACTOR

* Fix asyncio reactor test

* Docs: fix title

* Reword docs

* Check the TWISTED_REACTOR setting outside of the installing function

* Remove unrelated change

* Update scrapy/utils/log.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-06 22:42:34 +05:00
Mikhail Korobov 8b8df31961
Merge pull request #4227 from elacuesta/name-resolver 
Name resolver with IPv6 support
 2020-01-25 05:03:13 +05:00
Peter Vandenabeele 7d5cebcf77
fix logical documentation error with PER_DOMAIN or PER_DOMAIN 2020-01-23 09:08:21 +01:00
Eugenio Lacuesta b471765d40
[doc] FAQ entry about the IPv6 and the DNS_RESOLVER setting 2020-01-18 01:52:29 -03:00
Marc Hernandez Cabot c0d84f0962 fix typos 2019-12-18 19:39:21 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Mikhail Korobov a97a9b57bf
Merge pull request #3621 from Gallaecio/bfo-for-few-requests 
Document that the crawl order is BFO for small numbers of start requests
 2019-07-08 21:26:04 +05:00
Adrián Chaves 783de8a8d8 Document how request concurrency settings impact request order 2019-07-08 14:23:21 +02:00
Adrián Chaves 6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Konstantin Lopuhin bb1f311891 Add PyPy3 support to faq and install doc 2017-12-25 15:46:05 +03:00
Konstantin Lopuhin ea41114cf0 Mention PyPy support, add PyPy to install docs 2017-12-25 12:35:04 +03:00
Hugo 44623687ab Drop support for EOL Python 3.3 2017-12-19 17:59:05 +02:00
Aditya 9dd680d5c9 Use https for external links wherever possible in docs 2017-10-26 23:33:45 +05:30
Paul Tremberth 163618c9b7 FAQ Rewrite note on Python 3 support on Windows 2017-03-29 12:02:44 +02:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Pawel Miech b67440dec0 docs on Scrapy on Windows + Python 3 2016-07-04 16:37:51 +02:00
Pawel Miech 07d1605586 [docs] warnings about windows + python 3 in faq and install 2016-06-17 13:28:51 +02:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Konstantin Lopuhin 1ec49c2ada Fix FAQ entry about python versions support 2016-04-12 11:48:57 +03:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Ally Weir bd2fe996aa Spelling correction 
incorrect use of "too" instead of "to"
 2015-06-01 20:47:22 +05:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina 7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Pablo Hoffman 1a12922b77 improve scrapy deploy documentation 2015-04-09 17:39:01 -03:00
Shadab Zafar 5a58d64131 Fix some redirection links in documentation 
Fixes #606
 2015-03-18 19:41:26 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Daniel Graña 5b2faf61c3 recognize jl extension as jsonlines exporter and update docs 2014-06-25 13:55:15 -03:00
Julia Medina 80081054a2 Fix broken links in documentation 2014-04-09 18:57:52 -03:00
Capi Etheriel 72b6c96d9a Running lucasdemarchi/codespell to fix typos in docs 2014-03-06 12:40:55 -03:00
Daniel Graña 2df8156431 Drop Python 2.6 support 2013-10-29 13:44:00 -02:00
Hart 0ad01c34d4 fixed typo to parallel fix on 0.16 branch 2013-08-03 17:06:10 -07:00
cacovsky 8007762890 Add FAQ entry referencing Request.meta usage 2013-05-27 13:02:17 -03:00
Mikhail Korobov b245d592aa Update faq.rst 
spider.DOWNLOAD_DELAY is deprecated
 2013-04-18 02:42:15 +06:00
Pablo Hoffman bb20907254 minor updated to faq 2013-03-14 16:43:00 -03:00