RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,607 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

113 Commits

Author SHA1 Message Date
pankaj1707k c1bbb299d7
Add and run pre-commit hook 'blacken-docs' 
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
 2023-02-01 16:30:57 +05:30
NaincyKumariKnoldus e264cc30d1
removed the pywin32 docs section (#5370) 2022-03-10 14:54:33 +01:00
Andrey Rahmatullin a7bd6fa229
Merge pull request #3669 from Gallaecio/spider-name-collision 2021-08-17 19:42:53 +05:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Andrey Rahmatullin 954c4b48e6
Merge pull request #3667 from Gallaecio/long-allowed-domains 
Add a FAQ entry on how to deal with long lists of allowed domains
 2021-03-05 20:16:37 +05:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Adrián Chaves d29bec60d7
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest) (#4504) 
* Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest)

* Log a detailed error message to discover why MockServer is not working

* Go for all lines!

* Disable tests based on mitmproxy while running on PyPy

* Fix test_get_func_args for PyPy 3.6+

* Make testPayloadDefaultCiphers work regardless of OpenSSL default ciphers

* Crossing fingers…

* Rename: testPayloadDefaultCiphers → testPayloadDisabledCipher

* Test the PyPy version currently documented as the minimum required version

* Fix the PYPY_VERSION tag

* Update the documentation about supported PyPy versions

* Also test the latest 3.5 Python version with PyPy

* Fix the PYPY_VERSION value for the latest 3.5 version

* Use pinned dependencies for asyncio and PyPy tests against oldest supported Python versions

* Fix PyPy installation for the pypy3-pinned Tox environment

* Try installing Cython

* Maybe PyPy requires lxml 3.6.0?

* install.rst: minor clarification

* lxml 4.0.0 is required on PyPy

* Require setuptools 18.5+

* Revert "Require setuptools 18.5+"

This reverts commit 017ec33ac2.

* Maintain lxml as a dependency if setuptools < 18.5 is used
 2020-07-17 02:19:24 +05:00
Adrián Chaves 88ade19675
Merge branch 'master' into spider-name-collision 2020-06-18 14:01:02 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta 8b549392f9
Bump minimum Python version to 3.5.2 (#4615) 2020-06-11 14:53:59 +02:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Mikhail Korobov 14612fc3d3
Merge pull request #4534 from elacuesta/deprecate-baseitem 
Deprecate scrapy.item.BaseItem
 2020-05-16 03:24:06 +05:00
Adrián Chaves 36c3c9713e
Run tests with Python 3.5.0 (#4518) 
* Run tests with Python 3.5.0

* Use Ubuntu Trusty 14.04 to test Python 3.5.0

* Use mitmproxy<3.0.0 on Python 3.5.0

* Skip tests requiring mitmproxy in Python 3.5.0

* Change the minimum Python version from 3.5 to 3.5.1

* Do not prevent Scrapy from working with Python 3.5.0

* Force system Python 3.5.1

* Do not install a non-system Python in Python 3.5.1 jobs

* Switch to Trusty to be able to test Xenial’s Python version ¯\_(ツ)_/¯

* Add missing trusty

* Stop breaking old PyPy

* Allow installing Scrapy on Python 3.5.0
 2020-05-15 22:37:56 +05:00
Eugenio Lacuesta 7988c676a9
Update Item docstring, update BaseItem occurrences 2020-05-05 13:11:01 -03:00
Adrián Chaves ec0a0eb3ab
Merge branch 'master' into spider-name-collision 2020-04-28 21:13:08 +02:00
nyov a34c366fa4 DOC linkcheck run; https and 301 link updates. 
Closes #4359
 2020-02-25 16:51:33 +00:00
elacuesta 35dafef7f1
Specify Twisted reactor (TWISTED_REACTOR setting) (#4294) 
* Add the ability to install a specific reactor

* Add docs for the TWISTED_REACTOR setting

* Add tests for the TWISTED_REACTOR setting

* Update asyncio reactor test

* Ignore W503 globally

W503 is not PEP8-compliant:
c59c4376ad

* Line length adjustment

* Adjust asyncio reactor tests

* Merge ASYNCIO_ENABLED and TWISTED_REACTOR settings

* More docs about TWISTED_REACTOR

* Fix asyncio reactor test

* Docs: fix title

* Reword docs

* Check the TWISTED_REACTOR setting outside of the installing function

* Remove unrelated change

* Update scrapy/utils/log.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-06 22:42:34 +05:00
Mikhail Korobov 8b8df31961
Merge pull request #4227 from elacuesta/name-resolver 
Name resolver with IPv6 support
 2020-01-25 05:03:13 +05:00
Peter Vandenabeele 7d5cebcf77
fix logical documentation error with PER_DOMAIN or PER_DOMAIN 2020-01-23 09:08:21 +01:00
Eugenio Lacuesta b471765d40
[doc] FAQ entry about the IPv6 and the DNS_RESOLVER setting 2020-01-18 01:52:29 -03:00
Marc Hernandez Cabot c0d84f0962 fix typos 2019-12-18 19:39:21 +01:00
Adrián Chaves 57e36b5f0d
Merge branch 'master' into spider-name-collision 2019-12-05 13:03:43 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Mikhail Korobov a97a9b57bf
Merge pull request #3621 from Gallaecio/bfo-for-few-requests 
Document that the crawl order is BFO for small numbers of start requests
 2019-07-08 21:26:04 +05:00
Adrián Chaves 783de8a8d8 Document how request concurrency settings impact request order 2019-07-08 14:23:21 +02:00
Adrián Chaves 6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Adrián Chaves 72cf190145 Add a FAQ entry about name collisions 2019-03-22 15:37:01 +01:00
Adrián Chaves 120007c057 Add a FAQ entry on how to deal with long lists of allowed domains 2019-03-08 13:53:47 +01:00
Konstantin Lopuhin bb1f311891 Add PyPy3 support to faq and install doc 2017-12-25 15:46:05 +03:00
Konstantin Lopuhin ea41114cf0 Mention PyPy support, add PyPy to install docs 2017-12-25 12:35:04 +03:00
Hugo 44623687ab Drop support for EOL Python 3.3 2017-12-19 17:59:05 +02:00
Aditya 9dd680d5c9 Use https for external links wherever possible in docs 2017-10-26 23:33:45 +05:30
Paul Tremberth 163618c9b7 FAQ Rewrite note on Python 3 support on Windows 2017-03-29 12:02:44 +02:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Pawel Miech b67440dec0 docs on Scrapy on Windows + Python 3 2016-07-04 16:37:51 +02:00
Pawel Miech 07d1605586 [docs] warnings about windows + python 3 in faq and install 2016-06-17 13:28:51 +02:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Konstantin Lopuhin 1ec49c2ada Fix FAQ entry about python versions support 2016-04-12 11:48:57 +03:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Ally Weir bd2fe996aa Spelling correction 
incorrect use of "too" instead of "to"
 2015-06-01 20:47:22 +05:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00