RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,083 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

9083 Commits

Author SHA1 Message Date
Péter Ferenc Gyarmati 38d2a154ec docs: use https scheme for each quotes.toscrape.com url occurrence 2022-02-06 18:52:15 +01:00
Andrey Rahmatullin 3533717488
Merge pull request #5393 from elacuesta/remove-textresponse-body-as-unicode 
Remove deprecated TextResponse.body_as_unicode
 2022-02-06 19:36:55 +05:00
Eugenio Lacuesta 55ae2109c9
Remove deprecated TextResponse.body_as_unicode 2022-02-05 13:11:13 -03:00
Adrián Chaves 9be878fc09
CI: stop using tox-pip-version (#5389) 2022-02-04 12:27:39 +01:00
Laerte fe43411bc4
Fix TypeError on using pathlib.Path as key on FEEDS settings (#5384) 2022-02-04 09:57:57 +01:00
Sixuan (Cherie) Wu 30d5779ea9
Fix FEED_URI_PARAMS: custom params throws KeyError (#4966) 
* fix FEED_URI_PARAMS: custom params throws KeyError closes #4962

* another try FEED_URI_PARAMS

* add warning message and change default function

* Add tests for FEED_URI_PARAMS

* FEED_URI_PARAMS: warn if the params dict has been modified in-place

* [Doc] FEED_URI_PARAMS: modifying params in-place is deprecated

* Remove whileline

* Rename parameters for lambda function

* Type hints for FeedExporter._get_uri_params

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2022-01-28 14:30:30 -03:00
Georgiy Zatserklianyi 4bdaa54af4
response_httprepr memory issue fixed (#4972) 
* response_httprepr replaced by response.body

* unused import deleted

* get_header_size function added

* response size calculation updated

* flake8 codestyle fix

* added counting status code, line breaks to response size

* get_status size: list to tuple, comments added

* test added: comparing new response size counting method with old `len(response_httprepr)`

* downloader stats : unreachable code deleted

* `get_status_size` optimized

* comment added

* tests.test_downloadermiddleware_stats: statement formatting updated

* scrapy.utils.response: `response_httprepr` -> deprecated

* tests.test_downloadermiddleware_stats: flake8 fix
 2022-01-28 18:39:32 +05:00
Mikhail Korobov 9dd77b42b5
Merge pull request #1582 from starrify/add_doc_textresponse_urljoin 
added: Doc for `scrapy.http.TextResponse.urljoin`
 2022-01-23 21:40:21 +05:00
Andrey Rahmatullin 92764d68e2
Merge pull request #3155 from aprasanna/doc-fix 
A few typo fixes and some grammatical enhancements
 2022-01-21 13:59:03 +05:00
Andrey Rahmatullin 5059b8c493
Merge branch 'master' into doc-fix 2022-01-21 13:58:43 +05:00
Andrey Rahmatullin 169bc8866e
Merge pull request #4335 from leobalestri/patch-1 
Update install.rst
 2022-01-21 13:26:11 +05:00
Andrey Rahmatullin 1608b1a7fa
Merge branch 'master' into patch-1 2022-01-21 13:24:51 +05:00
Andrey Rahmatullin 9b8285d98a
Merge pull request #5334 from wRAR/mw-manager-typing-fix 
Fix typing of middleware methods.
 2022-01-10 11:33:12 +05:00
Andrey Rahmatullin b04cfa4832
Merge pull request #5352 from wRAR/crawler-process-reactor-later 2022-01-06 11:31:59 +05:00
Andrey Rakhmatullin 3577c00569 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2022-01-03 13:59:26 +05:00
Andrey Rahmatullin c5ab58056c
Set WindowsSelectorEventLoopPolicy on Windows (#5315) 2021-12-31 20:38:10 +01:00
Andrey Rakhmatullin 6eaceec735 Implement docs suggestions. 2021-12-31 20:14:24 +05:00
Andrey Rakhmatullin 60800ee5f2 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-31 20:12:30 +05:00
Andrey Rakhmatullin a2763c608d Remove unused MiddlewareManager._process_chain_both(). 2021-12-31 18:35:19 +05:00
Andrey Rakhmatullin a7846d2398 Merge remote-tracking branch 'origin/master' into mw-manager-typing-fix 2021-12-31 18:34:29 +05:00
Andrey Rahmatullin 57dc58123b
Remove the experimental note about asyncio (#5332) 2021-12-31 13:15:08 +01:00
Paweł Miech e4bdd1cb95
downloader.webclient: make reactor import local (#5357) 2021-12-31 11:57:12 +01:00
Andrey Oskin b81938684b
Docs: correct process repetition start step (#5356) 
The process repeats from step 3, the scheduler feeds request to the engine. Steps 1 and 2 are not parts of the loop as their incarnations steps 7 and 8 are parts of the loop.
 2021-12-31 11:49:18 +01:00
Andrey Rakhmatullin 0e78b68650 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-31 15:46:25 +05:00
Andrey Rakhmatullin 64261d9e38 Slight refactoring. 2021-12-31 15:45:59 +05:00
Andrey Rahmatullin 7380888cad
Fix a warning message. (#5359) 2021-12-30 10:55:16 -03:00
Andrey Rakhmatullin a986792def Add more docs for TWISTED_REACTOR. 2021-12-24 19:43:14 +05:00
Andrey Rakhmatullin 940cc0776f Add docs about TWISTED_REACTOR and other per-process settings. 2021-12-24 17:12:50 +05:00
Andrey Rakhmatullin d4565318c7 Fix a reactor test on Windows. 2021-12-23 17:40:31 +05:00
Andrey Rakhmatullin df04e934e6 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-23 17:25:56 +05:00
Andrey Rakhmatullin 9c4bfb4836 Remove an unused import. 2021-12-23 17:17:36 +05:00
Andrey Rakhmatullin 0025134382 Completely skip WindowsRunSpiderCommandTest outside Windows. 2021-12-23 16:45:17 +05:00
Andrey Rakhmatullin ebcafdf4a9 Add tests for TWISTED_REACTOR in custom_settings. 2021-12-23 16:35:26 +05:00
Andrey Rakhmatullin 041699b54c Remove tests that want to modify the test process reactor. 2021-12-23 16:14:47 +05:00
Andrey Rakhmatullin 60c8838554 Move installing the reactor from CrawlerProcess to Crawler. 2021-12-23 16:07:18 +05:00
Andrey Rakhmatullin 46ef9cf771 Don't install non-working shutdown handlers in `scrapy shell`. 2021-12-23 16:05:17 +05:00
yogender26 d6a384b3cf
corrrection of coma (#5347) 2021-12-22 23:39:05 +01:00
Andrey Rahmatullin 9ec60e8e4a
Merge pull request #5320 from zessx/5319-oib-base-tag 2021-12-21 17:49:31 +05:00
Andrey Rahmatullin 0862b179e3
Merge pull request #5333 from wRAR/pylint-2.12 
Fix and pin pylint.
 2021-12-16 18:54:27 +05:00
Andrey Rakhmatullin 6483dfdbe1 Move install_shutdown_handlers() from __init__() to start(). 2021-12-01 19:53:39 +05:00
Andrey Rakhmatullin eb62906c3e Extract utils.log.log_reactor_info(). 2021-12-01 17:40:41 +05:00
Andrey Rakhmatullin 4cc039628e Fix typing of middleware methods. 2021-11-26 19:52:03 +05:00
Andrey Rakhmatullin 6ec66c96fb Fix and pin pylint. 2021-11-26 12:25:45 +05:00
Alex c316ca45a5
Use augmented assignment statements (#5322) 2021-11-16 10:20:56 +01:00
Samuel Marchal 75ed765476 Test coverage for open_in_browser base tag injection (#5319) 2021-11-15 16:44:56 +01:00
Samuel Marchal f2c800c5c9 Improve open_in_browser base tag injection (#5319) 2021-11-15 16:44:56 +01:00
Andrey Rahmatullin 28eba610e2
Re-enable Windows tests for Python 3.9 and 3.10. (#5316) 2021-11-15 08:24:54 +01:00
Andrey Rahmatullin 9828cb3f0b
Merge pull request #5310 from azzamsa/more-test 
test: `test_format_engine_status`
 2021-11-08 20:13:00 +05:00
azzamsa 55cce25a79 test: `test_format_engine_status` 2021-11-05 03:02:02 +07:00
Andrey Rahmatullin ed50a81f50
Merge pull request #5299 from azzamsa/fix-tests 
fix: `CodeBlockParser` has been renamed to `PythonCodeBlockParser`
 2021-11-04 19:00:14 +05:00