RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,095 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

9095 Commits

Author SHA1 Message Date
Alex e2e2ffd0d1
Move from optparse to argparse (#5374) 2022-02-09 20:52:07 +01:00
Andrey Rahmatullin 1356836e7e
Merge pull request #5405 from wRAR/tests-twisted-22.1-fix 
Fix running tests with Twisted 22.1.0
 2022-02-09 00:06:25 +05:00
Andrey Rakhmatullin 77547a1ab5 Revert "Temporarily pin Twisted to an older version in CI (#5401)" 
This reverts commit b282a7af01.
 2022-02-08 21:06:02 +05:00
Andrey Rakhmatullin 1e1cfc26db Copy resource classes from twisted.web.test.test_webclient. 2022-02-08 21:01:16 +05:00
Raihan Nismara fd55f62207
Update Logo in README.rst (#5258) 2022-02-08 15:36:25 +01:00
Laerte 4bda0976b2
Fix csviter call, add parse_rows test (#5394) 2022-02-08 14:57:19 +01:00
Andrey Rahmatullin 840362496c
Merge pull request #5396 from peter-gy/master 
docs: use https scheme for each quotes.toscrape.com url occurrence
 2022-02-08 11:05:52 +05:00
Andrey Rahmatullin b282a7af01
Temporarily pin Twisted to an older version in CI (#5401) 2022-02-07 21:25:08 +01:00
Andrey Rahmatullin f7bf7414f0
Merge pull request #5398 from elacuesta/remove-deprecated-item-classes 
Remove deprecated DictItem/BaseItem classes
 2022-02-07 12:34:35 +05:00
Eugenio Lacuesta fca49cca92
Remove deprecated DictItem class 2022-02-06 18:31:55 -03:00
Eugenio Lacuesta c8c1edd43b
Flake8 adjustments 2022-02-06 18:27:41 -03:00
Eugenio Lacuesta bbfa185664
Remove deprecated BaseItem class 2022-02-06 18:12:28 -03:00
Péter Ferenc Gyarmati 38d2a154ec docs: use https scheme for each quotes.toscrape.com url occurrence 2022-02-06 18:52:15 +01:00
Andrey Rahmatullin 3533717488
Merge pull request #5393 from elacuesta/remove-textresponse-body-as-unicode 
Remove deprecated TextResponse.body_as_unicode
 2022-02-06 19:36:55 +05:00
Eugenio Lacuesta 55ae2109c9
Remove deprecated TextResponse.body_as_unicode 2022-02-05 13:11:13 -03:00
Adrián Chaves 9be878fc09
CI: stop using tox-pip-version (#5389) 2022-02-04 12:27:39 +01:00
Laerte fe43411bc4
Fix TypeError on using pathlib.Path as key on FEEDS settings (#5384) 2022-02-04 09:57:57 +01:00
Sixuan (Cherie) Wu 30d5779ea9
Fix FEED_URI_PARAMS: custom params throws KeyError (#4966) 
* fix FEED_URI_PARAMS: custom params throws KeyError closes #4962

* another try FEED_URI_PARAMS

* add warning message and change default function

* Add tests for FEED_URI_PARAMS

* FEED_URI_PARAMS: warn if the params dict has been modified in-place

* [Doc] FEED_URI_PARAMS: modifying params in-place is deprecated

* Remove whileline

* Rename parameters for lambda function

* Type hints for FeedExporter._get_uri_params

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2022-01-28 14:30:30 -03:00
Georgiy Zatserklianyi 4bdaa54af4
response_httprepr memory issue fixed (#4972) 
* response_httprepr replaced by response.body

* unused import deleted

* get_header_size function added

* response size calculation updated

* flake8 codestyle fix

* added counting status code, line breaks to response size

* get_status size: list to tuple, comments added

* test added: comparing new response size counting method with old `len(response_httprepr)`

* downloader stats : unreachable code deleted

* `get_status_size` optimized

* comment added

* tests.test_downloadermiddleware_stats: statement formatting updated

* scrapy.utils.response: `response_httprepr` -> deprecated

* tests.test_downloadermiddleware_stats: flake8 fix
 2022-01-28 18:39:32 +05:00
Mikhail Korobov 9dd77b42b5
Merge pull request #1582 from starrify/add_doc_textresponse_urljoin 
added: Doc for `scrapy.http.TextResponse.urljoin`
 2022-01-23 21:40:21 +05:00
Andrey Rahmatullin 92764d68e2
Merge pull request #3155 from aprasanna/doc-fix 
A few typo fixes and some grammatical enhancements
 2022-01-21 13:59:03 +05:00
Andrey Rahmatullin 5059b8c493
Merge branch 'master' into doc-fix 2022-01-21 13:58:43 +05:00
Andrey Rahmatullin 169bc8866e
Merge pull request #4335 from leobalestri/patch-1 
Update install.rst
 2022-01-21 13:26:11 +05:00
Andrey Rahmatullin 1608b1a7fa
Merge branch 'master' into patch-1 2022-01-21 13:24:51 +05:00
Andrey Rahmatullin 9b8285d98a
Merge pull request #5334 from wRAR/mw-manager-typing-fix 
Fix typing of middleware methods.
 2022-01-10 11:33:12 +05:00
Andrey Rahmatullin b04cfa4832
Merge pull request #5352 from wRAR/crawler-process-reactor-later 2022-01-06 11:31:59 +05:00
Andrey Rakhmatullin 3577c00569 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2022-01-03 13:59:26 +05:00
Andrey Rahmatullin c5ab58056c
Set WindowsSelectorEventLoopPolicy on Windows (#5315) 2021-12-31 20:38:10 +01:00
Andrey Rakhmatullin 6eaceec735 Implement docs suggestions. 2021-12-31 20:14:24 +05:00
Andrey Rakhmatullin 60800ee5f2 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-31 20:12:30 +05:00
Andrey Rakhmatullin a2763c608d Remove unused MiddlewareManager._process_chain_both(). 2021-12-31 18:35:19 +05:00
Andrey Rakhmatullin a7846d2398 Merge remote-tracking branch 'origin/master' into mw-manager-typing-fix 2021-12-31 18:34:29 +05:00
Andrey Rahmatullin 57dc58123b
Remove the experimental note about asyncio (#5332) 2021-12-31 13:15:08 +01:00
Paweł Miech e4bdd1cb95
downloader.webclient: make reactor import local (#5357) 2021-12-31 11:57:12 +01:00
Andrey Oskin b81938684b
Docs: correct process repetition start step (#5356) 
The process repeats from step 3, the scheduler feeds request to the engine. Steps 1 and 2 are not parts of the loop as their incarnations steps 7 and 8 are parts of the loop.
 2021-12-31 11:49:18 +01:00
Andrey Rakhmatullin 0e78b68650 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-31 15:46:25 +05:00
Andrey Rakhmatullin 64261d9e38 Slight refactoring. 2021-12-31 15:45:59 +05:00
Andrey Rahmatullin 7380888cad
Fix a warning message. (#5359) 2021-12-30 10:55:16 -03:00
Andrey Rakhmatullin a986792def Add more docs for TWISTED_REACTOR. 2021-12-24 19:43:14 +05:00
Andrey Rakhmatullin 940cc0776f Add docs about TWISTED_REACTOR and other per-process settings. 2021-12-24 17:12:50 +05:00
Andrey Rakhmatullin d4565318c7 Fix a reactor test on Windows. 2021-12-23 17:40:31 +05:00
Andrey Rakhmatullin df04e934e6 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-23 17:25:56 +05:00
Andrey Rakhmatullin 9c4bfb4836 Remove an unused import. 2021-12-23 17:17:36 +05:00
Andrey Rakhmatullin 0025134382 Completely skip WindowsRunSpiderCommandTest outside Windows. 2021-12-23 16:45:17 +05:00
Andrey Rakhmatullin ebcafdf4a9 Add tests for TWISTED_REACTOR in custom_settings. 2021-12-23 16:35:26 +05:00
Andrey Rakhmatullin 041699b54c Remove tests that want to modify the test process reactor. 2021-12-23 16:14:47 +05:00
Andrey Rakhmatullin 60c8838554 Move installing the reactor from CrawlerProcess to Crawler. 2021-12-23 16:07:18 +05:00
Andrey Rakhmatullin 46ef9cf771 Don't install non-working shutdown handlers in `scrapy shell`. 2021-12-23 16:05:17 +05:00
yogender26 d6a384b3cf
corrrection of coma (#5347) 2021-12-22 23:39:05 +01:00
Andrey Rahmatullin 9ec60e8e4a
Merge pull request #5320 from zessx/5319-oib-base-tag 2021-12-21 17:49:31 +05:00