RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,393 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

10393 Commits

Author SHA1 Message Date
Georgiy Zatserklianyi 4bdaa54af4
response_httprepr memory issue fixed (#4972) 
* response_httprepr replaced by response.body

* unused import deleted

* get_header_size function added

* response size calculation updated

* flake8 codestyle fix

* added counting status code, line breaks to response size

* get_status size: list to tuple, comments added

* test added: comparing new response size counting method with old `len(response_httprepr)`

* downloader stats : unreachable code deleted

* `get_status_size` optimized

* comment added

* tests.test_downloadermiddleware_stats: statement formatting updated

* scrapy.utils.response: `response_httprepr` -> deprecated

* tests.test_downloadermiddleware_stats: flake8 fix
 2022-01-28 18:39:32 +05:00
Mikhail Korobov 9dd77b42b5
Merge pull request #1582 from starrify/add_doc_textresponse_urljoin 
added: Doc for `scrapy.http.TextResponse.urljoin`
 2022-01-23 21:40:21 +05:00
Andrey Rahmatullin 92764d68e2
Merge pull request #3155 from aprasanna/doc-fix 
A few typo fixes and some grammatical enhancements
 2022-01-21 13:59:03 +05:00
Andrey Rahmatullin 5059b8c493
Merge branch 'master' into doc-fix 2022-01-21 13:58:43 +05:00
Andrey Rahmatullin 169bc8866e
Merge pull request #4335 from leobalestri/patch-1 
Update install.rst
 2022-01-21 13:26:11 +05:00
Andrey Rahmatullin 1608b1a7fa
Merge branch 'master' into patch-1 2022-01-21 13:24:51 +05:00
Andrey Rakhmatullin e079bffdab Disable logging-fstring-interpolation in pylint. 2022-01-11 19:21:07 +05:00
Andrey Rakhmatullin f789547551 Implement spider middleware iterable upgrade/downgrade. 2022-01-11 18:28:32 +05:00
Andrey Rakhmatullin 8864407c4d Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2022-01-10 19:37:46 +05:00
Andrey Rahmatullin 9b8285d98a
Merge pull request #5334 from wRAR/mw-manager-typing-fix 
Fix typing of middleware methods.
 2022-01-10 11:33:12 +05:00
Andrey Rahmatullin b04cfa4832
Merge pull request #5352 from wRAR/crawler-process-reactor-later 2022-01-06 11:31:59 +05:00
Andrey Rakhmatullin 3577c00569 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2022-01-03 13:59:26 +05:00
Andrey Rahmatullin c5ab58056c
Set WindowsSelectorEventLoopPolicy on Windows (#5315) 2021-12-31 20:38:10 +01:00
Andrey Rakhmatullin 6eaceec735 Implement docs suggestions. 2021-12-31 20:14:24 +05:00
Andrey Rakhmatullin 60800ee5f2 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-31 20:12:30 +05:00
Andrey Rakhmatullin a2763c608d Remove unused MiddlewareManager._process_chain_both(). 2021-12-31 18:35:19 +05:00
Andrey Rakhmatullin a7846d2398 Merge remote-tracking branch 'origin/master' into mw-manager-typing-fix 2021-12-31 18:34:29 +05:00
Andrey Rahmatullin 57dc58123b
Remove the experimental note about asyncio (#5332) 2021-12-31 13:15:08 +01:00
Paweł Miech e4bdd1cb95
downloader.webclient: make reactor import local (#5357) 2021-12-31 11:57:12 +01:00
Andrey Oskin b81938684b
Docs: correct process repetition start step (#5356) 
The process repeats from step 3, the scheduler feeds request to the engine. Steps 1 and 2 are not parts of the loop as their incarnations steps 7 and 8 are parts of the loop.
 2021-12-31 11:49:18 +01:00
Andrey Rakhmatullin 0e78b68650 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-31 15:46:25 +05:00
Andrey Rakhmatullin 64261d9e38 Slight refactoring. 2021-12-31 15:45:59 +05:00
Andrey Rahmatullin 7380888cad
Fix a warning message. (#5359) 2021-12-30 10:55:16 -03:00
Burak Can Kahraman a9dfd85ea6 Document coroutines for signals. 2021-12-30 15:48:53 +03:00
Andrey Rakhmatullin a986792def Add more docs for TWISTED_REACTOR. 2021-12-24 19:43:14 +05:00
Andrey Rakhmatullin 940cc0776f Add docs about TWISTED_REACTOR and other per-process settings. 2021-12-24 17:12:50 +05:00
Andrey Rakhmatullin d4565318c7 Fix a reactor test on Windows. 2021-12-23 17:40:31 +05:00
Andrey Rakhmatullin df04e934e6 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-23 17:25:56 +05:00
Andrey Rakhmatullin 9c4bfb4836 Remove an unused import. 2021-12-23 17:17:36 +05:00
Andrey Rakhmatullin 0025134382 Completely skip WindowsRunSpiderCommandTest outside Windows. 2021-12-23 16:45:17 +05:00
Andrey Rakhmatullin ebcafdf4a9 Add tests for TWISTED_REACTOR in custom_settings. 2021-12-23 16:35:26 +05:00
Andrey Rakhmatullin 041699b54c Remove tests that want to modify the test process reactor. 2021-12-23 16:14:47 +05:00
Andrey Rakhmatullin 60c8838554 Move installing the reactor from CrawlerProcess to Crawler. 2021-12-23 16:07:18 +05:00
Andrey Rakhmatullin 46ef9cf771 Don't install non-working shutdown handlers in `scrapy shell`. 2021-12-23 16:05:17 +05:00
yogender26 d6a384b3cf
corrrection of coma (#5347) 2021-12-22 23:39:05 +01:00
Andrey Rahmatullin 9ec60e8e4a
Merge pull request #5320 from zessx/5319-oib-base-tag 2021-12-21 17:49:31 +05:00
Andrey Rahmatullin 0862b179e3
Merge pull request #5333 from wRAR/pylint-2.12 
Fix and pin pylint.
 2021-12-16 18:54:27 +05:00
Andrey Rakhmatullin 6483dfdbe1 Move install_shutdown_handlers() from __init__() to start(). 2021-12-01 19:53:39 +05:00
Andrey Rakhmatullin eb62906c3e Extract utils.log.log_reactor_info(). 2021-12-01 17:40:41 +05:00
Andrey Rakhmatullin 4cc039628e Fix typing of middleware methods. 2021-11-26 19:52:03 +05:00
Andrey Rakhmatullin 6ec66c96fb Fix and pin pylint. 2021-11-26 12:25:45 +05:00
GeorgeA92 8185aa5265 per slot settings: codestyle fix 2021-11-20 21:06:08 +02:00
GeorgeA92 1b7d7ecfcd per slot settings: logging updated (create/close Slot) 2021-11-20 21:05:24 +02:00
GeorgeA92 d5acf88ca5 per slot settings: logging added (create Slot) 2021-11-20 20:14:28 +02:00
GeorgeA92 eea56c4912 per slot settings: creation of Downloader.Slot objects from per slot settings added 2021-11-20 20:13:54 +02:00
Alex c316ca45a5
Use augmented assignment statements (#5322) 2021-11-16 10:20:56 +01:00
Samuel Marchal 75ed765476 Test coverage for open_in_browser base tag injection (#5319) 2021-11-15 16:44:56 +01:00
Samuel Marchal f2c800c5c9 Improve open_in_browser base tag injection (#5319) 2021-11-15 16:44:56 +01:00
Andrey Rahmatullin 28eba610e2
Re-enable Windows tests for Python 3.9 and 3.10. (#5316) 2021-11-15 08:24:54 +01:00
Andrey Rahmatullin 9828cb3f0b
Merge pull request #5310 from azzamsa/more-test 
test: `test_format_engine_status`
 2021-11-08 20:13:00 +05:00