We're storing exceptions captured by Twisted on the media pipeline
cache, but we're also using the defer.returnValue method with our
own methods decorated with @defer.inlineCallbacks.
The defer.returnValue method passes returned values forward by
throwing a defer._DefGen_Return exception, which in its turn
extends the BaseException class and is captured by Twisted.
This way, the latest exception stored in the Failure's object may
also have an HtmlResponse object in its __context__ attribute. As
the Response object also keeps track of the Request object that
has originated it, you could figure it out how many RAM we're
wasting here.
This could easily lead to a Memory Leak problem when running
spiders with Media Pipeline enabled and a particular Request set
that tends to raise a significant number of exceptions.
Example triggers:
- media requests with 404 status responses
- user land exceptins coming from custom middlewares
- etc.
add doctest for function
no need in this variables
move common assertion inside function
rename variable
rename variables
rename function
use function
this is not a method of public API
correct name for test
Update docs/topics/settings.rst
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
This reverts commit 646164fd7d6dd52061804d2df7424cff929bf739.
remove tests
Revert "emit new signal"
This reverts commit fcde0c6880678957a76af6083b6248f430a00fcf.
Revert "documentation for new signal"
This reverts commit 8aeb9f696ece95c16499a96767a7afa3d9c4abf4.
rename variable
remove old write function
remove unused imports
remove old read function
remove unused function
use mock methods
mock downloader
close downloader
add parse method
use new PQ class
create mock downloader
use downloader directly
remove mark/unmark mechanism
update signature
documentation for new signal
utilize new signal
correct signal handler signature
emit new signal
test another signal
new signal
rename test file
faster test
rename test case
tests for signal emitting in bad cases
[wip] refactoring
* SlotPriorityQueues doesn't care about objects inside, it is now just
a container for multiple priority queues
* assorted variable renames
* don't inherit DownloaderAwarePriorityQueue from SlotBasedPriorityQueue
* apply @whalebot-helmsman's suggestions for __slots__ and meta issues
more bike-shedding
* remove mutable default arguments
* more verbose variable names
remove unneeded code
* PriorityAsTupleQueue.is_empty does the same as len(self) == 0
* custom PriorityAsTupleQueue.close is not needed after a switch
to namedtuples
* is_new and is_empty return values are unused
* "url" local variable is unused
PrioritySlot.__str__ shouldn't return unicode in Python 2
also, do some bike-shedding: _pathable -> _path_safe
use namedtuple for PrioritySlot
cleanup: _get_from_request does the same here
Request.meta is always a dict
Mikhail Korobov
Eugenio Lacuesta
Victor Torres
Daniel Graña
Anubhav Patel
sortafreel
Claudio Salazar
Andrew Baxter
Adrián Chaves
Matthijs Vos
Maram Sumanth
Kostis Anagnostopoulos
Konstantin Lopuhin
Vostretsov Nikita
Lucy Wang