RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,890 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

7890 Commits

Author SHA1 Message Date
nyov a34c366fa4 DOC linkcheck run; https and 301 link updates. 
Closes #4359
 2020-02-25 16:51:33 +00:00
Mikhail Korobov caa1dea890
Merge pull request #4370 from Gallaecio/dont-fail-kw-only 
Make BaseItemExporter’s dont_fail parameter keyword-only
 2020-02-25 13:12:01 +05:00
Adrián Chaves 7417a9871c Make BaseItemExporter’s dont_fail parameter keyword-only 2020-02-24 13:28:15 +01:00
elacuesta 31f35c9c00
Remove unnecessary comma (#4369) 2020-02-24 12:02:00 +01:00
Mikhail Korobov 49480225d7
Merge pull request #4365 from noviluni/fix_flake8_E701_E271 
Fix flake8 E131, E211, E251, E271, E701
 2020-02-23 15:52:52 +05:00
elacuesta 9d983c1b99
Expose certificate for HTTPS responses (#4054) 
* Expose certificate for HTTPS responses

* Fix test (missing inlineCallbacks decorator)

* Note about Response.certificate

* Explicitly cover None as the default value of Response.certificate

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-22 17:20:31 +05:00
Vostretsov Nikita 67ee0b097f
Remove specifics of downstream request queues from scheduler (#3884) 
* move serialization/deserialization logic to downstream queues

* make memory queues conform to common interface

* make ScrapyPriorityQueue conform common interface

* ScrapyPriorityQueue works with disk

* make key as string

* return list instead of dict as earlier

* downloader aware pq works with new interface

* we don`t need these methods anymore

* create directories for files

* remove dummy priority

* remove priority as parameter, let every queue decide for itself

* rename obj to request

* DownloaderAwarePriorityQueue is too thin wrapper around _SlotPriorityQueues, just remove second one

* remove priority as parameter, let every queue decide for itself

* rename argument

* more granular class separation

* python2 compatible

* one more argument for common interface

* more simple downstream queue interface

* single place for easier customization

* rename function

* shorter

* shorter

* use named arguments

* fix typo

* add docstring

* Update scrapy/pqueues.py

Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>

* Update scrapy/pqueues.py

Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>

* 4 spaces indentation

* we ok with existing directories

* remove unused import

* rename method

* remove unused imports

* it has no sense now

* relining

* note about queues

* add value

* Revert "it has no sense now"

This reverts commit b61604275b.

* pep8 E261

* pep8 E303

* pep8 E501

* pep8 E123

* pep8 E123

* use create instance

* remove excessive import

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-02-22 17:02:57 +05:00
Marc Hernandez Cabot 6e8e117aee fix flake E211 2020-02-22 12:32:38 +01:00
Marc Hernandez Cabot 69a8648bef fix E251 2020-02-22 12:32:38 +01:00
Marc Hernandez Cabot 9ad10bb6f7 fix E131 2020-02-22 12:32:38 +01:00
Marc Hernandez Cabot b49ece0b87 fix E701 and E271 flake8 2020-02-22 12:32:38 +01:00
Mikhail Korobov e551bc5170
Merge pull request #4318 from wRAR/asyncio-parse-asyncgen 
Support yield in async def callbacks.
 2020-02-21 23:50:01 +05:00
Mikhail Korobov b0a7efb88f
Merge pull request #4360 from noviluni/fix_flake8_E22X 
fix E22X flake8
 2020-02-21 23:45:27 +05:00
Mikhail Korobov 1be1101fe4
Merge pull request #4362 from noviluni/delete_deprecated_functions 
delete old deprecated functions from scrapy.utils.python
 2020-02-21 23:44:44 +05:00
Marc Hernandez Cabot 03ed9e1786 delete old deprecated functions from scrapy.utils.python 2020-02-21 09:29:29 +01:00
Marc Hernandez Cabot 6fb85951ce fix E22X flake8 2020-02-21 08:39:14 +01:00
Marc Hernández 91bbc70bc1
fix E30X flake8 (#4355) 2020-02-21 06:05:31 +01:00
Mikhail Korobov c4ee4b6075
Merge pull request #4347 from noviluni/deprecate_sel_shortcut 
Remove deprecated `sel` shortcut in scrapy shell
 2020-02-20 02:56:34 +05:00
Adrián Chaves 0f78a591f8
Fix Flake8-reported “Too many blank lines” 2020-02-19 19:09:39 +01:00
Adrián Chaves 6972a19707
Remove unused imports 2020-02-19 18:59:09 +01:00
Andrey Rahmatullin 88179027de
Merge pull request #4331 from Gallaecio/response-cb-kwargs 
Implement Response.cb_kwargs
 2020-02-19 22:40:14 +05:00
Marc Hernandez Cabot eb21dae524 deprecare sel shortcut in scrapy shell 2020-02-19 17:49:42 +01:00
Andrey Rahmatullin f558df2558
Merge pull request #4188 from elacuesta/logformatter-error-formatting 
LogFormatter error formatting
 2020-02-19 19:05:08 +05:00
Andrey Rahmatullin 528b894f28
Merge pull request #4321 from Gallaecio/link-extractor-encoding 
Use safe_url_string in link extraction
 2020-02-19 18:19:21 +05:00
Akshay Sharma 182445f9d9
Fix a spelling error: ie. → i.e. (#4338) 2020-02-18 17:58:31 +01:00
Mikhail Korobov 320cea62ff
Merge pull request #4309 from Gallaecio/virtualenv-doc 
Update installation instructions regarding Python 3 and virtual environments
 2020-02-18 19:56:35 +05:00
Adrián Chaves a04dd13cd0
ie. → i.e. 2020-02-14 22:31:30 +01:00
Adrián Chaves 5ae3e1678f
ie. → i.e. 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-14 22:30:36 +01:00
Adrián Chaves 43b43654a1 Add tests for meta and cb_kwargs not being available 2020-02-13 22:39:58 +01:00
Adrián Chaves 5ff9eb90ea Add a test for the copy of cb_kwargs from Request to Response 2020-02-13 22:36:18 +01:00
Adrián Chaves df937d8280 Implement Response.cb_kwargs 2020-02-13 22:33:36 +01:00
Adrián Chaves b4958358e8 Update tests to account for link extractors escaping spaces 2020-02-12 19:00:04 +01:00
Drew Seibert 2d6d4fb233
Deprecate overriding settings with SCRAPY-prefixed environment variables (#4300) 2020-02-11 10:35:23 +01:00
Mikhail Korobov a6ef065eb5
Merge pull request #4271 from wRAR/asyncio-signals 
async def support for signal handlers that already supported Deferreds
 2020-02-11 02:05:45 +05:00
Adrián Chaves 61e74bac76 Extract links with safe_url_string 
canonicalize_url changes links in undesirable ways.
 2020-02-10 21:57:21 +01:00
Andrey Rakhmatullin 1f0f52cbf7 Improve async signal tests. 2020-02-11 01:05:45 +05:00
Andrey Rakhmatullin 36dcf90184 Also test non-default async callbacks. 2020-02-11 01:00:16 +05:00
Abhishek Pratap Singh 4626e90df8
Allow updating flags in follow and follow_all (#4279) 2020-02-10 19:48:31 +01:00
Adrián Chaves 35723d76c0 Use canonicalize_url in link extraction 2020-02-07 22:59:53 +01:00
Adrián Chaves 4c6993f264
Merge branch 'master' into asyncio-parse-asyncgen 2020-02-07 21:34:54 +01:00
Adrián Chaves 59653ebac6 Update installation instructions regarding Python 3 and virtual environments 2020-02-07 21:07:57 +01:00
Mikhail Korobov b0eaf114e5
Merge pull request #4197 from elacuesta/sphinx-twisted-api 
[Docs] Fix Twisted links
 2020-02-07 23:51:15 +05:00
Mikhail Korobov bd7780277c
Merge pull request #4275 from abhishekh2001/master 
Fixed artwork/README formatting
 2020-02-07 23:44:15 +05:00
Mikhail Korobov 7e341e0f6b
Merge pull request #4291 from seregaxvm/master 
add zsh -h autocomplete option
 2020-02-07 23:42:10 +05:00
Mikhail Korobov c3b690a5b5
Merge pull request #4290 from dekimsey/patch-1 
FilesPipeline.file_path has optional arguments
 2020-02-07 23:41:31 +05:00
Mikhail Korobov 957681bcfa
Merge pull request #4272 from elacuesta/spider-middleware 
Spider middleware: catch spider callback exceptions early
 2020-02-07 23:40:50 +05:00
Mikhail Korobov afbaf9d430
Merge pull request #4303 from whalebot-helmsman/request_left_downloader_signal 
request_left_downloader signal
 2020-02-07 23:33:51 +05:00
Mikhail Korobov 0f62e44def
Merge pull request #4316 from wRAR/asyncio-parse-request-tests 
Add a test for an async callbacks that returns requests.
 2020-02-07 23:22:19 +05:00
Andrey Rakhmatullin 7323780c97 Support yield in async def callbacks. 2020-02-07 21:32:45 +05:00
Andrey Rakhmatullin 31f6c7112f Add a test for an async callbacks that returns requests. 2020-02-07 17:14:52 +05:00