* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
* Switch TestEngineBase to mockserver.
* Split mockserver.py.
* Stop using WrappingFactory.
* Remove keyfile/certfile fields from TestHttpBase.
* Move webclient-only resources to test_webclient.py.
* Refactor MockServer.url().
* Migrate TestHttpBase to mockserver.
* Migrate TestSimpleHttpsBase to a separate mockserver.
* Rename TestHttpMockServerBase.
* Migrate TestHttpProxyBase to a separate mockserver.
* Refactor HTTP mockservers.
* Migrate to pytest-twisted (WIP)
* Some typing fixes.
* Make --reactor=asyncio the default again.
* Try installing the correct event loop policy in tests on Windows.
* Make reactor_pytest a normal fixture.
* Fix test warnings.
* Fix FTPDownloadHandler teardown.
* Cleanups, typing.
* More cleanup.
* Update only_asyncio/only_not_asyncio mark messages.
* Switch to pytest.raises().
* Add matches= to broad pytest.raises().
* Adjust the test_nonserializable_object() regex for Python <= 3.11.
* Adjust the test_nonserializable_object() regex for PyPy.
* Adjust other test exception regexes for PyPy.
* Cleanup.
* Fix the display name of documented fingerprinter class methods
* Initial draft for the Scrapy 2.7 release notes
* Update VERSION and PREVIOUS_VERSION references
* Clarify the restrictions lifted for item field output names
* Fix the description of the BOM bug fix
* Fix the note about changes in MIME sniffing
* Fix typo
* Extend highlights
* Fyx typo
Obsolete REDIRECT_MAX_METAREFRESH_DELAY
which has been deprecated since Scrapy 0.18
Obsolete LOG_UNSERIALIZABLE_REQUESTS
which has been deprecated since Scrapy 1.2.0
and is replaced by SCHEDULER_DEBUG
add doctest for function
no need in this variables
move common assertion inside function
rename variable
rename variables
rename function
use function
this is not a method of public API
correct name for test
Update docs/topics/settings.rst
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
rename variable
remove old write function
remove unused imports
remove old read function
remove unused function
use mock methods
mock downloader
close downloader
add parse method
use new PQ class
create mock downloader
use downloader directly
remove mark/unmark mechanism
update signature
documentation for new signal
utilize new signal
correct signal handler signature
emit new signal
test another signal
new signal
rename test file
faster test
rename test case
tests for signal emitting in bad cases
[wip] refactoring
* SlotPriorityQueues doesn't care about objects inside, it is now just
a container for multiple priority queues
* assorted variable renames
* don't inherit DownloaderAwarePriorityQueue from SlotBasedPriorityQueue
* apply @whalebot-helmsman's suggestions for __slots__ and meta issues
more bike-shedding
* remove mutable default arguments
* more verbose variable names
remove unneeded code
* PriorityAsTupleQueue.is_empty does the same as len(self) == 0
* custom PriorityAsTupleQueue.close is not needed after a switch
to namedtuples
* is_new and is_empty return values are unused
* "url" local variable is unused
PrioritySlot.__str__ shouldn't return unicode in Python 2
also, do some bike-shedding: _pathable -> _path_safe
use namedtuple for PrioritySlot
cleanup: _get_from_request does the same here
Request.meta is always a dict
integration testing only everything is working, not logic of PQ
use method
create slot attribute in constructor
corect class for test case
stop crawler in teardown method
use class
correct entity naming
python 2 adaptation
integration test with crawler and spider
PEP8 fixes
no need to close implicitly
do not use pytest
need to put it into class
remove round-robin queue
additional check for empty queue
use pytest tmpdir fixture
handle exception
use O(N) instead of O(NlogN)
here we have request as struct
additional check for meptiness
small performance improvement
do not consume another request
test number of responses
mark requests
back to 3 slots test case
raise exceptions in case of missed meta
add marks to requests and work only with your own requests
only disk queue should obtain signals
separate functions for slot rasd/write
use signlas without variable
stop crawler
get signals in correct place
logic test for download-aware priority queue
update comment for structure
ensure text type
transform slot name to path
use implicit structure
use unicode type implicitly
use real crawler
add signals
more slot accounting
simple implementation of pop
small slot accounting code
no need for custom len function
ability to call super in py27
add slots
generic tests for downloader aware queue
dummy implementation of crawler aware priority queue
move common logic to base class
rename class
pass crawler to pqclass constructor
do not copy quelib.PriorityQueue code
add comment about new class
remove obsolete function
modify behaviour of queuelib.PriorityQueue to dodge very complex priority
better way to get name
remove obsolete commentary
check boundaries
function for priority convertion with known limits
correct import path
move file
do not switch on by deffault as ip concurrency not supported
set scheduler slot in case of empty slot
use constant
single place for added urls
single place for constants
use as default queue
correct format for error text
test migration from old version with on disk queue
in these tests we have only two inflection points - jobdir and priority_queue_cls
we do not need separate mock spider, use usual one
do not rely on order of dict elements, imply order of list
test round robiness of priority queue
add comments and requirements for our magick function
remove debug logging
put queues into slot
as we fabricate priorities we do not need special types anymore
fabricate priority for priority queue
more versatile priorities
Scheduler class is not inflection point
wrap correct types
check for emptinees before initialization
tests for new priority queue
correct default type for startprios
use exact values
put common settings to base class
test priorities for disk scheduler
test dequeue for disk scheduler
test length for disk scheduler
setUp/tearDown methods for on disk schedulers
new methods
remove excessive line
base class to handle scheduler creation
correct method names
test priorities
deque test
close scheduler on test end
enqueue some requests
test template for scheduler
use downloader slot
I/O implementation for RoundRobin queue
round-robin implementation without I/O and slot detection
wrappers for every disk queue class
Andrey Rakhmatullin
Laerte Pereira
Adrián Chaves
Sanchay Kumar
Laerte Pereira
Jalil SA
Emmanuel Rondan
Mikhail Korobov
Marc Hernández
Lukas Anzinger
Eugenio Lacuesta
elacuesta
nyov
Lucy Wang
Vostretsov Nikita