Obsolete REDIRECT_MAX_METAREFRESH_DELAY
which has been deprecated since Scrapy 0.18
Obsolete LOG_UNSERIALIZABLE_REQUESTS
which has been deprecated since Scrapy 1.2.0
and is replaced by SCHEDULER_DEBUG
* fix issue 4175 - Scrapy does not use a non-zero exit code when pipeline's open_spider throws the exception
* remove extra blank lines
* remove redundant code
* remove blank line at end of file
* more suitable naming for response and make if-condition shorter
* avoid error - AttributeError: 'Deferred' object has no attribute 'result'
* use getattr to make code concisely
* add test
* remove useless file
* modify test class name
* remove unneccessary files
* Fix Flake8-reported issue
* fix these items which are suggested by Gallaecio
・Sort those imports at tests/test_cmdline_crawl_with_pipeline/__init__.py
・Remove the unused setUp method.
・Remove comments generated by Scrapy’s project generation tool.
・Remove the [deploy] section from the scrapy.cfg file (I don’t think it’s needed here)
・Remove BOT_NAME and NEWSPIDER_MODULE from settings.py (I think there are not needed either, although I’m less sure about NEWSPIDER_MODULE)
* have to reserve BOT_NAME, SPIDER_MODULES in settings.py
* Remove unneeded empty lines
* Empty __init__.py file with unneeded comments
* Remove an unneeded empty line at the end
* Remove unneeed empty line from __init__.py file
* Update __init__.py
* Update __init__.py
* Update exception.py
* Update normal.py
* Update __init__.py
* Update __init__.py
* fix W391 blank line at end of file
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Expose certificate for HTTPS responses
* Fix test (missing inlineCallbacks decorator)
* Note about Response.certificate
* Explicitly cover None as the default value of Response.certificate
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* move serialization/deserialization logic to downstream queues
* make memory queues conform to common interface
* make ScrapyPriorityQueue conform common interface
* ScrapyPriorityQueue works with disk
* make key as string
* return list instead of dict as earlier
* downloader aware pq works with new interface
* we don`t need these methods anymore
* create directories for files
* remove dummy priority
* remove priority as parameter, let every queue decide for itself
* rename obj to request
* DownloaderAwarePriorityQueue is too thin wrapper around _SlotPriorityQueues, just remove second one
* remove priority as parameter, let every queue decide for itself
* rename argument
* more granular class separation
* python2 compatible
* one more argument for common interface
* more simple downstream queue interface
* single place for easier customization
* rename function
* shorter
* shorter
* use named arguments
* fix typo
* add docstring
* Update scrapy/pqueues.py
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
* Update scrapy/pqueues.py
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
* 4 spaces indentation
* we ok with existing directories
* remove unused import
* rename method
* remove unused imports
* it has no sense now
* relining
* note about queues
* add value
* Revert "it has no sense now"
This reverts commit b61604275b.
* pep8 E261
* pep8 E303
* pep8 E501
* pep8 E123
* pep8 E123
* use create instance
* remove excessive import
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
Eugenio Lacuesta
nyov
Adrián Chaves
sakshamb2113
gunblues
Mikhail Korobov
elacuesta
Vostretsov Nikita
Marc Hernandez Cabot
Andrey Rahmatullin
Abhishek Pratap Singh
Lane Shaw