RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

191 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 11d7a05a6f
Deprecate is_secure=False for s3://. (#7738) 2026-07-20 14:59:11 +02:00
Andrey Rakhmatullin 61f99f2df1
More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
Adrian 9523e1ec8c
Merge commit from fork 2026-07-07 12:25:36 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Kailesh 86a7ceaa9f
Dynamic loading for S3 HTTPS handler (#7370) 2026-03-30 10:12:43 +02:00
Andrey Rakhmatullin ccfa052fa1
Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
Andrey Rakhmatullin 11977afba5
Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
Andrey Rakhmatullin 3186ccf5d5
Reactorless tests 1: the test env (#6952) 
* Reactorless tests.

* Updates after merging master.

* Update a new import.

* Updates after merging master.

* Update testenv:no-reactor-pinned.

* Foundations for the reactorless mode.

* Fixes after merging master.

* Updates after merging reactorless-base.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Fix a test.

* Fixes.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.

* Fix a new import.

* Clarify --reactor marks.

* Cleanup.

* More docs/comments.

* Update test_fallback_workflow().

* Update TestDeferredFFromCoroF.

* Typing improvements.
 2026-02-02 19:25:30 +05:00
Andrey Rakhmatullin d1bd8eb49f
Async API for download handlers. (#7164) 2025-12-31 19:03:16 +01:00
Andrey Rakhmatullin 552f2fb91e
Generic improvements related to reactorless tests (#6968) 
* Skip doctests that import the reactor.

* Extract FTPDownloadHandler tests.

* Skip TestHttps2ClientProtocol via pytestmark.

* Replace some explicit reactor.callLater() in tests.

* Simplify TestRequestSendOrder._test_request_order().
 2025-07-25 13:35:00 +05:00
Andrey Rakhmatullin e225d0dea4
Ban more imports that import twisted.internet.reactor. (#6947) 2025-07-10 21:19:50 +05:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938) 
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
 2025-07-06 21:27:17 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin 8b3c3ea4ae
Refactor test functions that were returning a Deferred directly. (#6889) 2025-06-30 09:19:23 +02:00
Andrey Rakhmatullin b4d11b8b25
Further reduce deps on unittest. (#6884) 2025-06-11 01:28:09 +02:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Andrey Rakhmatullin 8ae418df44
Rewrite download handler tests to coroutines. (#6846) 2025-06-02 16:02:08 +02:00
Andrey Rakhmatullin 916fe50974
Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
Andrey Rakhmatullin b41aea4873
Restructure download handler tests. (#6821) 
* Restructure download handler tests.

* Typo.

* Use mixins to reduce boilerplate.
 2025-05-27 18:19:47 +05:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin 380c2279b9
Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00
Andrey Rakhmatullin ba5df629a2
Refactor downloader tests (#6647) 
* Make download handler test base classes abstract.

* Small cleanup.

* Don't run the full test suite for special HTTP cases.

* Don't run tests in imported base classes.

* Remove an obsolete service_identity check.

* Move FTP imports back to the top level.

* Simplify the H2DownloadHandler import.

* Forbig pytest 8.2.x.

* Revert "Simplify the H2DownloadHandler import."

This reverts commit ed187046ac.
 2025-02-03 20:11:47 +05:00
Andrey Rakhmatullin 783b98deda Make mockserver instances per-class. 2025-02-02 14:10:09 +05:00
Andrey Rakhmatullin 0a80871c3a Remove scrapy.core.downloader.webclient._parse(). 2025-01-28 22:22:09 +05:00
Andrey Rakhmatullin 16b998f9ca Sort out webclient tests. 2025-01-28 01:35:04 +05:00
Andrey Rakhmatullin c87354cd46 Enable SIM Ruff rules. 2025-01-02 01:03:36 +05:00
Andrey Rakhmatullin 273620488c Enable PTH Ruff rules. 2025-01-01 22:21:51 +05:00
Andrey Rakhmatullin 7dd92e6e43
Add pylint rules to ruff, refresh the ignore list of pylint itself (#6584) 2024-12-16 09:44:36 +01:00
Andrey Rakhmatullin e7595837a6 Add flake8-raise rules to ruff. 2024-12-12 20:33:35 +05:00
Andrey Rakhmatullin d85c39f5bc
Deprecation removals. (#6500) 
* Deprecation removals.

* Clean up the default pytest filterwarnings.

* Remove test_get_images_old().

* Redo boto-requiring test filtering.

* Remove an unused function.

* Improve the Crawler.crawl() error message.

* Fix the test.
 2024-10-31 18:06:22 +05:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin 3d8dbd5648 flake8-bugbear 2024-06-20 00:27:38 +05:00
Jon b0ef9a89a1
Remove usage of deprecated mktemp (#5285) 2024-02-26 19:53:06 +01:00
Swati Anshu c67f730695
create_instance → build_from_crawler, build_from_settings (#6169) 2023-12-18 11:51:02 +01:00
Laerte Pereira 09ce0ef526 Remove test_download_gzip_response test 2023-11-23 10:50:46 -03:00
Andrey Rakhmatullin fb4debda04 Use path_to_file_uri to fix the test on Windows. 2023-10-03 16:47:03 +04:00
Andrey Rakhmatullin d3b5c9be97 Deprecate running Crawler.crawl() twice. 2023-09-06 19:29:44 +04:00
freddiewanah c4f0aa4fdf
Refactor test cases to improve unit test quality (#5986) 2023-07-26 13:09:03 +02:00
Marc Hernández 5360ba34bc
IOError and other cleanup (#4716) 2023-06-21 20:08:53 +02:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Andrey Rakhmatullin f85c3f3d68 Add a comment about `tmpname + '^'`. 2022-11-25 18:46:14 +06:00
Laerte Pereira e769532644
Remove `noconnect` deprecate code 2022-11-23 08:30:11 -03:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Mikhail Korobov 26c70318cb make Scrapy testing suite more robust in environments where non-existing hosts are resolvable 2022-07-17 16:47:20 +05:00
Kromitvs 9e265a2c1f
Mind body to choose response class in cache, FTP and HTTP/1.0 (#4873) 2022-06-16 20:52:19 +02:00