Commit Graph

52 Commits

Author SHA1 Message Date
Adrian Chaves 184166d629 Use get_* for getters 2026-06-29 10:58:18 +02:00
Adrian Chaves 5aae8334a5 Improve request and scope delay support 2026-06-26 19:00:06 +02:00
Adrian Chaves 8caa1ac4db Fix typing issues 2026-06-26 13:27:14 +02:00
Adrian Chaves c889c1057d Improve test coverage 2026-06-26 13:09:17 +02:00
Adrian Chaves 6ce90dde3e Minimum queuelib: 1.4.2 → 1.6.1 2026-06-26 12:15:52 +02:00
Adrian Chaves 8190b5b703 Add a throttling-aware scheduler 2026-06-26 10:44:31 +02:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Andrey Rakhmatullin fc14a0ce59
Enable mypy warn_return_any. (#7492) 2026-05-04 22:24:47 +02:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259)
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin ccfa052fa1
Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
Andrey Rakhmatullin 11977afba5
Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
Andrey Rakhmatullin c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. (#7233)
* Move no-reactor to pytest-asyncio.

* Unify the style.

* Skip a test that installs a reactor in no-reactor.

* Fix pinned env deps.

* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
Andrey Rakhmatullin 3186ccf5d5
Reactorless tests 1: the test env (#6952)
* Reactorless tests.

* Updates after merging master.

* Update a new import.

* Updates after merging master.

* Update testenv:no-reactor-pinned.

* Foundations for the reactorless mode.

* Fixes after merging master.

* Updates after merging reactorless-base.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Fix a test.

* Fixes.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.

* Fix a new import.

* Clarify --reactor marks.

* Cleanup.

* More docs/comments.

* Update test_fallback_workflow().

* Update TestDeferredFFromCoroF.

* Typing improvements.
2026-02-02 19:25:30 +05:00
Andrey Rakhmatullin 57f539d7a5
Deprecate Crawler.stop(). (#6999) 2025-08-11 08:32:55 +02:00
Andrey Rakhmatullin 8c8f4ff033
Extract in-process test HTTP servers into separate processes (#6960)
* Switch TestEngineBase to mockserver.

* Split mockserver.py.

* Stop using WrappingFactory.

* Remove keyfile/certfile fields from TestHttpBase.

* Move webclient-only resources to test_webclient.py.

* Refactor MockServer.url().

* Migrate TestHttpBase to mockserver.

* Migrate TestSimpleHttpsBase to a separate mockserver.

* Rename TestHttpMockServerBase.

* Migrate TestHttpProxyBase to a separate mockserver.

* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938)
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
Laerte Pereira dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting (#6921)
* Deprecate CONCURRENT_REQUESTS_PER_IP setting

* Typing fix

* PR review

* tests

* oops

* Update scrapy/settings/default_settings.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* update logic

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-06-30 16:51:05 +05:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin 7bbe775040
Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680)
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin dc706d4fc3 Remove useless pylint: disable lines. 2025-01-02 12:32:25 +05:00
Andrey Rakhmatullin c2832ed131 Add flake8-pyi rules to ruff. 2024-12-12 20:44:28 +05:00
Andrey Rakhmatullin 04d0411bf7
Filter test-time warnings. (#6501) 2024-10-21 12:30:49 +02:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Andrey Rakhmatullin 1c70d3e605 flake8-comprehensions 2024-06-19 23:36:36 +05:00
Sanchay Kumar c9ef520936
Add Downloader.get_slot_key() without a spider parameter (#6352) 2024-05-10 14:26:45 +02:00
Laerte Pereira 26a16f2c43 Fix tests 2024-02-28 16:36:19 -03:00
Andrey Rakhmatullin b4acf5c827
Fix and remove most of the entries from the mypy ignore list (#6137) 2023-11-07 09:34:35 +01:00
Andrey Rakhmatullin bb15c93a2b Add Crawler._load_settings(). 2023-09-12 14:44:16 +04:00
Andrey Rakhmatullin 036d5836d0 Move more things from Crawler.__init__() to .crawl(). 2023-09-08 19:29:05 +04:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Adrián Chaves 06c8f673af
2.7 release notes (#5680)
* Fix the display name of documented fingerprinter class methods

* Initial draft for the Scrapy 2.7 release notes

* Update VERSION and PREVIOUS_VERSION references

* Clarify the restrictions lifted for item field output names

* Fix the description of the BOM bug fix

* Fix the note about changes in MIME sniffing

* Fix typo

* Extend highlights

* Fyx typo
2022-10-17 18:04:29 +05:00
Mikhail Korobov b103664bf4
Address 2/3 of warnings from tests (#5561) 2022-07-19 17:39:26 +02:00
Andrey Rahmatullin 8284de5e76
Fix/silence the Pylint messages added in 2.10 (#5235) 2021-08-24 12:15:29 +02:00
Marc Hernández 336f19f5cc
Change super syntax (#4707) 2020-08-04 20:42:01 +02:00
Lukas Anzinger 3672f5f988 Spider constructor expects name as argument, not start_urls.
Fixes #4644
2020-06-23 14:56:12 +02:00
Eugenio Lacuesta 8643e8d355
Flake8: remove E123 (Closing bracket does not match indentation of opening bracket's line) 2020-05-06 12:26:04 -03:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
nyov 1b591ff061 Obsolete deprecated settings
Obsolete REDIRECT_MAX_METAREFRESH_DELAY
  which has been deprecated since Scrapy 0.18

Obsolete LOG_UNSERIALIZABLE_REQUESTS
  which has been deprecated since Scrapy 1.2.0
  and is replaced by SCHEDULER_DEBUG
2020-03-03 14:03:30 +00:00
Lucy Wang df574de8cc improve tests and fix some lint warnings (#6)
* refactor downloader-aware test cases

* fix lint

* add doctest for _path_safe

* remove unused code

* better doctest
2019-03-25 16:08:40 +03:00
Vostretsov Nikita 8afffb7234 Tests Cleanup
add doctest for function
no need in this variables
move common assertion inside function
rename variable
rename variables
rename function
use function
this is not a method of public API
correct name for test
Update docs/topics/settings.rst

Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
2019-03-25 16:08:40 +03:00
Vostretsov Nikita 443fb98a47 Use downloader directly
rename variable
remove old write function
remove unused imports
remove old read function
remove unused function
use mock methods
mock downloader
close downloader
add parse method
use new PQ class
create mock downloader
use downloader directly
remove mark/unmark mechanism
2019-03-25 16:08:40 +03:00
Vostretsov Nikita 3b1db71dac New signal
update signature
documentation for new signal
utilize new signal
correct signal handler signature
emit new signal
test another signal
new signal
rename test file
faster test
rename test case
tests for signal emitting in bad cases
2019-03-25 16:08:40 +03:00
Vostretsov Nikita 757f53a324 Address Lucy's comments
add tests to check correctness of slot setermination
unmark requests after downloading
shorter
better exception message
2019-03-25 16:08:40 +03:00
Mikhail Korobov 90934959d0 actually apply __slots__ suggestion
[wip] refactoring

* SlotPriorityQueues doesn't care about objects inside, it is now just
  a container for multiple priority queues
* assorted variable renames
* don't inherit DownloaderAwarePriorityQueue from SlotBasedPriorityQueue
* apply @whalebot-helmsman's suggestions for __slots__ and meta issues
more bike-shedding

* remove mutable default arguments
* more verbose variable names
remove unneeded code

* PriorityAsTupleQueue.is_empty does the same as len(self) == 0
* custom PriorityAsTupleQueue.close is not needed after a switch
  to namedtuples
* is_new and is_empty return values are unused
* "url" local variable is unused
PrioritySlot.__str__ shouldn't return unicode in Python 2

also, do some bike-shedding: _pathable -> _path_safe
use namedtuple for PrioritySlot
cleanup: _get_from_request does the same here

Request.meta is always a dict
2019-03-25 16:08:40 +03:00
Vostretsov Nikita 987c2ae4a9 test ip concurrency incompatibility with DAPQ 2019-03-22 15:40:37 +03:00
Vostretsov Nikita d970be64cc Integration test
integration testing only everything is working, not logic of PQ
use method
create slot attribute in constructor
corect class for test case
stop crawler in teardown method
use class
correct entity naming
python 2 adaptation
integration test with crawler and spider
2019-03-22 15:40:36 +03:00