You could define a spider attribute that references a callback method
but if this method has a different name than your spider attribute,
the request serializer is not able to find it on the spider class.
With this commit we're fixing this behavior as we're searching for
callback references in the spider object itself instead of looking
for attributes with the same function's name, that could be different.
Nones in allowed_domains ought to be ignored and there are also tests
for that scenario. This commit fixes the handling of None and also the
accompanying tests which are now executed again.
Obsolete REDIRECT_MAX_METAREFRESH_DELAY
which has been deprecated since Scrapy 0.18
Obsolete LOG_UNSERIALIZABLE_REQUESTS
which has been deprecated since Scrapy 1.2.0
and is replaced by SCHEDULER_DEBUG
* fix issue 4175 - Scrapy does not use a non-zero exit code when pipeline's open_spider throws the exception
* remove extra blank lines
* remove redundant code
* remove blank line at end of file
* more suitable naming for response and make if-condition shorter
* avoid error - AttributeError: 'Deferred' object has no attribute 'result'
* use getattr to make code concisely
* add test
* remove useless file
* modify test class name
* remove unneccessary files
* Fix Flake8-reported issue
* fix these items which are suggested by Gallaecio
・Sort those imports at tests/test_cmdline_crawl_with_pipeline/__init__.py
・Remove the unused setUp method.
・Remove comments generated by Scrapy’s project generation tool.
・Remove the [deploy] section from the scrapy.cfg file (I don’t think it’s needed here)
・Remove BOT_NAME and NEWSPIDER_MODULE from settings.py (I think there are not needed either, although I’m less sure about NEWSPIDER_MODULE)
* have to reserve BOT_NAME, SPIDER_MODULES in settings.py
* Remove unneeded empty lines
* Empty __init__.py file with unneeded comments
* Remove an unneeded empty line at the end
* Remove unneeed empty line from __init__.py file
* Update __init__.py
* Update __init__.py
* Update exception.py
* Update normal.py
* Update __init__.py
* Update __init__.py
* fix W391 blank line at end of file
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Expose certificate for HTTPS responses
* Fix test (missing inlineCallbacks decorator)
* Note about Response.certificate
* Explicitly cover None as the default value of Response.certificate
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* move serialization/deserialization logic to downstream queues
* make memory queues conform to common interface
* make ScrapyPriorityQueue conform common interface
* ScrapyPriorityQueue works with disk
* make key as string
* return list instead of dict as earlier
* downloader aware pq works with new interface
* we don`t need these methods anymore
* create directories for files
* remove dummy priority
* remove priority as parameter, let every queue decide for itself
* rename obj to request
* DownloaderAwarePriorityQueue is too thin wrapper around _SlotPriorityQueues, just remove second one
* remove priority as parameter, let every queue decide for itself
* rename argument
* more granular class separation
* python2 compatible
* one more argument for common interface
* more simple downstream queue interface
* single place for easier customization
* rename function
* shorter
* shorter
* use named arguments
* fix typo
* add docstring
* Update scrapy/pqueues.py
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
* Update scrapy/pqueues.py
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
* 4 spaces indentation
* we ok with existing directories
* remove unused import
* rename method
* remove unused imports
* it has no sense now
* relining
* note about queues
* add value
* Revert "it has no sense now"
This reverts commit b61604275b.
* pep8 E261
* pep8 E303
* pep8 E501
* pep8 E123
* pep8 E123
* use create instance
* remove excessive import
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
Eugenio Lacuesta
Mikhail Korobov
Victor Torres
Eugenio Lacuesta
Adrián Chaves
Eugenio Lacuesta
Andrey Rahmatullin
Lukas Anzinger
nyov
nyov
sakshamb2113
gunblues
Vostretsov Nikita
Marc Hernandez Cabot