Commit Graph

132 Commits

Author SHA1 Message Date
Adrian c5ec881f1d
Update tools (#7724) 2026-07-07 17:27:26 +05:00
Andrey Rakhmatullin 4e956bd2de
Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs (#7575)
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.

* Update the docs.

* Trim the tables.

* Restore lost wording.

* Handlers docs improvements and fixes.
2026-06-09 01:10:44 +05:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373)
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
2026-04-09 16:56:30 +05:00
Kailesh a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER (#7361) 2026-04-08 13:58:35 +02:00
Andrey Rakhmatullin ed63fa94d6
Deprecate spider args of middlewares and pipelines. (#7006)
* Deprecate spider args of spider middleware methods.

* Deprecate the spider arg of pipeline process_item().

* Fix a typing issue.

* Deprecate the spider arg of pipeline {open,close}_spider().

* Cleanup.

* Update docs.

* Add pragma: no cover to some deprecated code.

* More tests.

* More tests.

* More tests.

* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Andrey Rakhmatullin ce5a132f12
Run and fix linkcheck. (#6524) 2024-11-04 11:40:07 +01:00
Julian Ste c8e87ab21a
Fixed typos (#6497) 2024-10-17 17:03:16 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Kevin Toms 9074c16497 make suggestion 2024-01-18 09:36:25 -05:00
Kevin Toms 46f94ec9cb Fix test
Wrap the yield line in a function to prevent throwing error when the code snippet is executed
2024-01-17 15:49:51 -05:00
Kevin Toms 88285e75b6 Add FAQ on making a blank request 2024-01-17 10:05:22 -05:00
Adrián Chaves 150d96764b Deprecate xmliter in favor of xmliter_lxml 2023-12-15 11:42:55 +01:00
DeflateAwning 87c8c51999
Fix a typo in the FAQ (#5904) 2023-04-20 08:23:02 +02:00
Andrey Rakhmatullin 76eba9977b
Skip or fix failing code block tests (#5826) 2023-02-14 09:40:38 +01:00
pankaj1707k 03f32c018f
Update black version and fix minor typos and punctuations 2023-02-03 11:52:20 +05:30
pankaj1707k c1bbb299d7
Add and run pre-commit hook 'blacken-docs'
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
2023-02-01 16:30:57 +05:30
NaincyKumariKnoldus e264cc30d1
removed the pywin32 docs section (#5370) 2022-03-10 14:54:33 +01:00
Andrey Rahmatullin a7bd6fa229
Merge pull request #3669 from Gallaecio/spider-name-collision 2021-08-17 19:42:53 +05:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Andrey Rahmatullin 954c4b48e6
Merge pull request #3667 from Gallaecio/long-allowed-domains
Add a FAQ entry on how to deal with long lists of allowed domains
2021-03-05 20:16:37 +05:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512)
Co-authored-by: Yuval Hager <yhager@yhager.com>
2020-08-17 15:10:08 +02:00
Adrián Chaves d29bec60d7
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest) (#4504)
* Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest)

* Log a detailed error message to discover why MockServer is not working

* Go for all lines!

* Disable tests based on mitmproxy while running on PyPy

* Fix test_get_func_args for PyPy 3.6+

* Make testPayloadDefaultCiphers work regardless of OpenSSL default ciphers

* Crossing fingers…

* Rename: testPayloadDefaultCiphers → testPayloadDisabledCipher

* Test the PyPy version currently documented as the minimum required version

* Fix the PYPY_VERSION tag

* Update the documentation about supported PyPy versions

* Also test the latest 3.5 Python version with PyPy

* Fix the PYPY_VERSION value for the latest 3.5 version

* Use pinned dependencies for asyncio and PyPy tests against oldest supported Python versions

* Fix PyPy installation for the pypy3-pinned Tox environment

* Try installing Cython

* Maybe PyPy requires lxml 3.6.0?

* install.rst: minor clarification

* lxml 4.0.0 is required on PyPy

* Require setuptools 18.5+

* Revert "Require setuptools 18.5+"

This reverts commit 017ec33ac2.

* Maintain lxml as a dependency if setuptools < 18.5 is used
2020-07-17 02:19:24 +05:00
Adrián Chaves 88ade19675
Merge branch 'master' into spider-name-collision 2020-06-18 14:01:02 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta 8b549392f9
Bump minimum Python version to 3.5.2 (#4615) 2020-06-11 14:53:59 +02:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Mikhail Korobov 14612fc3d3
Merge pull request #4534 from elacuesta/deprecate-baseitem
Deprecate scrapy.item.BaseItem
2020-05-16 03:24:06 +05:00
Adrián Chaves 36c3c9713e
Run tests with Python 3.5.0 (#4518)
* Run tests with Python 3.5.0

* Use Ubuntu Trusty 14.04 to test Python 3.5.0

* Use mitmproxy<3.0.0 on Python 3.5.0

* Skip tests requiring mitmproxy in Python 3.5.0

* Change the minimum Python version from 3.5 to 3.5.1

* Do not prevent Scrapy from working with Python 3.5.0

* Force system Python 3.5.1

* Do not install a non-system Python in Python 3.5.1 jobs

* Switch to Trusty to be able to test Xenial’s Python version ¯\_(ツ)_/¯

* Add missing trusty

* Stop breaking old PyPy

* Allow installing Scrapy on Python 3.5.0
2020-05-15 22:37:56 +05:00
Eugenio Lacuesta 7988c676a9
Update Item docstring, update BaseItem occurrences 2020-05-05 13:11:01 -03:00
Adrián Chaves ec0a0eb3ab
Merge branch 'master' into spider-name-collision 2020-04-28 21:13:08 +02:00
nyov a34c366fa4 DOC linkcheck run; https and 301 link updates.
Closes #4359
2020-02-25 16:51:33 +00:00
elacuesta 35dafef7f1
Specify Twisted reactor (TWISTED_REACTOR setting) (#4294)
* Add the ability to install a specific reactor

* Add docs for the TWISTED_REACTOR setting

* Add tests for the TWISTED_REACTOR setting

* Update asyncio reactor test

* Ignore W503 globally

W503 is not PEP8-compliant:
c59c4376ad

* Line length adjustment

* Adjust asyncio reactor tests

* Merge ASYNCIO_ENABLED and TWISTED_REACTOR settings

* More docs about TWISTED_REACTOR

* Fix asyncio reactor test

* Docs: fix title

* Reword docs

* Check the TWISTED_REACTOR setting outside of the installing function

* Remove unrelated change

* Update scrapy/utils/log.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
2020-02-06 22:42:34 +05:00
Mikhail Korobov 8b8df31961
Merge pull request #4227 from elacuesta/name-resolver
Name resolver with IPv6 support
2020-01-25 05:03:13 +05:00
Peter Vandenabeele 7d5cebcf77
fix logical documentation error with PER_DOMAIN or PER_DOMAIN 2020-01-23 09:08:21 +01:00
Eugenio Lacuesta b471765d40
[doc] FAQ entry about the IPv6 and the DNS_RESOLVER setting 2020-01-18 01:52:29 -03:00
Marc Hernandez Cabot c0d84f0962 fix typos 2019-12-18 19:39:21 +01:00
Adrián Chaves 57e36b5f0d
Merge branch 'master' into spider-name-collision 2019-12-05 13:03:43 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892)
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
2019-08-07 12:36:52 +05:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Mikhail Korobov a97a9b57bf
Merge pull request #3621 from Gallaecio/bfo-for-few-requests
Document that the crawl order is BFO for small numbers of start requests
2019-07-08 21:26:04 +05:00
Adrián Chaves 783de8a8d8 Document how request concurrency settings impact request order 2019-07-08 14:23:21 +02:00
Adrián Chaves 6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Adrián Chaves 72cf190145 Add a FAQ entry about name collisions 2019-03-22 15:37:01 +01:00
Adrián Chaves 120007c057 Add a FAQ entry on how to deal with long lists of allowed domains 2019-03-08 13:53:47 +01:00