RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,080 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

1744 Commits

Author SHA1 Message Date
Laerte fe43411bc4
Fix TypeError on using pathlib.Path as key on FEEDS settings (#5384) 2022-02-04 09:57:57 +01:00
Sixuan (Cherie) Wu 30d5779ea9
Fix FEED_URI_PARAMS: custom params throws KeyError (#4966) 
* fix FEED_URI_PARAMS: custom params throws KeyError closes #4962

* another try FEED_URI_PARAMS

* add warning message and change default function

* Add tests for FEED_URI_PARAMS

* FEED_URI_PARAMS: warn if the params dict has been modified in-place

* [Doc] FEED_URI_PARAMS: modifying params in-place is deprecated

* Remove whileline

* Rename parameters for lambda function

* Type hints for FeedExporter._get_uri_params

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2022-01-28 14:30:30 -03:00
Georgiy Zatserklianyi 4bdaa54af4
response_httprepr memory issue fixed (#4972) 
* response_httprepr replaced by response.body

* unused import deleted

* get_header_size function added

* response size calculation updated

* flake8 codestyle fix

* added counting status code, line breaks to response size

* get_status size: list to tuple, comments added

* test added: comparing new response size counting method with old `len(response_httprepr)`

* downloader stats : unreachable code deleted

* `get_status_size` optimized

* comment added

* tests.test_downloadermiddleware_stats: statement formatting updated

* scrapy.utils.response: `response_httprepr` -> deprecated

* tests.test_downloadermiddleware_stats: flake8 fix
 2022-01-28 18:39:32 +05:00
Andrey Rakhmatullin 3577c00569 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2022-01-03 13:59:26 +05:00
Andrey Rahmatullin c5ab58056c
Set WindowsSelectorEventLoopPolicy on Windows (#5315) 2021-12-31 20:38:10 +01:00
Andrey Rakhmatullin d4565318c7 Fix a reactor test on Windows. 2021-12-23 17:40:31 +05:00
Andrey Rakhmatullin df04e934e6 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-23 17:25:56 +05:00
Andrey Rakhmatullin 9c4bfb4836 Remove an unused import. 2021-12-23 17:17:36 +05:00
Andrey Rakhmatullin 0025134382 Completely skip WindowsRunSpiderCommandTest outside Windows. 2021-12-23 16:45:17 +05:00
Andrey Rakhmatullin ebcafdf4a9 Add tests for TWISTED_REACTOR in custom_settings. 2021-12-23 16:35:26 +05:00
Andrey Rakhmatullin 041699b54c Remove tests that want to modify the test process reactor. 2021-12-23 16:14:47 +05:00
Andrey Rakhmatullin 60c8838554 Move installing the reactor from CrawlerProcess to Crawler. 2021-12-23 16:07:18 +05:00
Andrey Rahmatullin 9ec60e8e4a
Merge pull request #5320 from zessx/5319-oib-base-tag 2021-12-21 17:49:31 +05:00
Alex c316ca45a5
Use augmented assignment statements (#5322) 2021-11-16 10:20:56 +01:00
Samuel Marchal 75ed765476 Test coverage for open_in_browser base tag injection (#5319) 2021-11-15 16:44:56 +01:00
azzamsa 55cce25a79 test: `test_format_engine_status` 2021-11-05 03:02:02 +07:00
Andrey Rahmatullin 144d1eb834
Add Deferred-to-Future helpers (#5288) 2021-10-22 18:46:01 +02:00
Adrián Chaves d774d6a9c4
Remove unused variable 2021-10-15 17:25:22 +02:00
Erik Kemperman ca320feb2a
Add LOG_FILE_APPEND to settings 2021-10-15 16:36:41 +02:00
Andrey Rahmatullin 5b13bfd14a
Merge pull request #5269 from Ankur19/allow-non-text-response-types-in-ItemLoader 
Fix bug #5145 - Removing Selector for Response's that are not Http or Xml
 2021-10-14 16:19:54 +05:00
Kian-Meng, Ang 3a263280ba Fix typos 2021-10-11 22:32:42 +08:00
ankur19 6fbd6f941f Fix issue#5145 
Fix condition for failing tests

set Selector to None on AttributeError

Add test and remove unused imports

Fix imports
 2021-10-10 15:55:22 -04:00
Georgiy Zatserklianyi d3f1bf79e8
Use f-strings where appropriate (#5246) 2021-10-07 16:27:20 +02:00
Eugenio Lacuesta 029cab72e8
[CI] fix pypy test (#5264) 2021-10-06 19:34:09 +02:00
Laerte 6c858cec91
Cookies: Cast primitive types to str (#5253) 
* cast primitive types to str

* add tests
 2021-10-06 12:32:04 -03:00
Andrey Rakhmatullin b081f18a2f Add http_auth_domain to HttpAuthMiddleware. 2021-10-06 12:33:18 +02:00
Eugenio Lacuesta 890f884de4
Allow 'callback' key in keyword arguments for request callbacks (#5251) 2021-10-01 09:50:42 +02:00
Andrey Rahmatullin 8284de5e76
Fix/silence the Pylint messages added in 2.10 (#5235) 2021-08-24 12:15:29 +02:00
D R Siddhartha 43ea21e830
Feed post-processing plugin support (#5190) 2021-08-24 11:48:01 +02:00
Matsievskiy S.V 3f635eb683
Extract domain from genspider URL (#4439) 2021-08-24 11:05:50 +02:00
Andrey Rahmatullin ada539a63a
Merge pull request #4676 from Gallaecio/startproject-existing-folder 
Fix startproject failing for existing folders
 2021-08-24 12:18:49 +05:00
Andrey Rahmatullin 731f2d3085
Merge pull request #4178 from elacuesta/remove_spider_make_requests_from_url 2021-08-20 16:16:25 +05:00
Andrey Rahmatullin 2814e0e197
Disable builtin middlewares in spider middleware tests. (#5229) 2021-08-16 13:22:01 +02:00
Andrey Rakhmatullin b63369c148 Rename tests/requirements-py3.txt to tests/requirements.txt. 2021-08-11 20:02:45 +05:00
Andrey Rahmatullin 0590c3756c
Merge pull request #5221 from wRAR/3.10-tests 
Fixing tests for upcoming 3.10.
 2021-08-11 14:19:51 +05:00
Andrey Rakhmatullin 983b89ad4f Fix SpiderLoaderTest on Python 3.10. 2021-08-11 10:39:23 +05:00
Michel Ace ce9d6c658b
Add more rel_has_nofollow tests 2021-08-10 22:21:51 +02:00
Michel Ace 295f0e2bf5
Make flake8 happy 2021-08-10 21:38:29 +02:00
Michel Ace 07d20a8ce4
Fix test_rel_has_nofollow test 2021-08-10 21:21:43 +02:00
Michel Ace 18b6f30a73
Add test for rel_has_nofollow 2021-08-10 21:13:50 +02:00
Mannan2812 94baa4b272
Fix FileFeedStoragePreFeedOptionsTest fails in CI/CD pipeline (#5198) 2021-08-05 21:23:11 +02:00
Andrey Rakhmatullin 93bf1ae7e3 Fix tests for the 3.10 TypeError message change. 2021-08-03 20:16:59 +05:00
Andrey Rahmatullin b406affd5b
Merge pull request #5210 from laggardkernel/feature/aws-session-token 2021-07-30 15:15:55 +05:00
Eugenio Lacuesta cc89f6be38
Response.attributes (#5218) 2021-07-29 22:12:44 +02:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
Pascal Corpet 70dddfe2b2 Typing: switch to a newer version of MyPy to check types 2021-07-21 17:10:10 +02:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Iván de Prado eca641aa3d
Update tests/test_engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:20 +01:00
Iván de Prado Alonso f35970778b Test case for raising CloseSpider on spider idle signal handler 2021-06-29 13:21:38 +01:00