RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

2349 Commits

Author SHA1 Message Date
Chenwei Niu 1f797d0fdb
Removed some deprecated functions and functionalities (#6116) 2023-10-30 09:59:11 +01:00
Adrián Chaves 1d81585612
Merge pull request #6112 from wRAR/test-shutdown-forced 
Make shutdown tests more robust.
 2023-10-30 09:35:29 +01:00
Andrey Rakhmatullin 6b0c18e921 Improve signal sending in test_shutdown_forced. 2023-10-19 23:13:47 +04:00
Andrey Rakhmatullin cc9c415bf3 Disable ipython for InteractiveShellTest. 2023-10-18 19:30:57 +04:00
Laerte Pereira 38dbd43993 Remove tests not necessary anymore 2023-10-18 06:29:47 -03:00
Laerte Pereira 644ab3af48 Remove support for feed storage backends without feed_options 2023-10-17 22:39:09 -03:00
Laerte Pereira 3db438127c Remove support for HttpCompressionMiddleware subclasses without stats 2023-10-17 21:38:38 -03:00
Laerte Pereira a2b9351f04 Remove support for using HttpAuthMiddleware without http_auth_domain 2023-10-17 20:20:38 -03:00
Laerte Pereira 2de95f1fc0 Remove leftover from _make_compatible removal 2023-10-17 19:44:12 -03:00
Laerte Pereira ad4e8b64d4 fix test_spider_manager_verify_interface test 2023-10-17 18:58:27 -03:00
Laerte Pereira aa0a425826 Remove deprecated code moved to itemloaders 2023-10-17 18:20:12 -03:00
Laerte Pereira 7a5cefbcfa Remove deprecated scrapy.downloadermiddlewares.decompression 2023-10-17 17:49:22 -03:00
Andrey Rakhmatullin 2cb1e10c76 Make shutdown tests more robust. 2023-10-17 23:56:54 +04:00
Andrey Rakhmatullin 991121fa91
Re-enable uvloop tests on 3.12 (#6098) 2023-10-17 19:24:44 +02:00
Andrey Rakhmatullin 5807970a22
Add typing hints to CaselessDict and Headers. (#6097) 2023-10-17 19:07:13 +02:00
Andrey Rakhmatullin 029a56384d Increase the timeout. 2023-10-09 00:16:41 +04:00
Andrey Rakhmatullin 9910ccf3ae Merge remote-tracking branch 'origin/master' into signals-proper 2023-10-08 23:02:11 +04:00
Andrey Rakhmatullin 2f436c05e0 Use SIGBREAK in Windows tests. 2023-10-08 22:55:05 +04:00
Andrey Rakhmatullin a338873e3a Re-enable bpython on Python 3.12. 2023-10-03 16:50:52 +04:00
Andrey Rakhmatullin fb4debda04 Use path_to_file_uri to fix the test on Windows. 2023-10-03 16:47:03 +04:00
Andrey Rakhmatullin 1ed9ed4f92 Require pyftpdlib that supports Python 3.12. 2023-10-02 21:33:02 +04:00
Andrey Rakhmatullin eb5e2e79ba Use SIGINT instead of SIGTERM to support Windows. 2023-09-24 23:31:46 +04:00
Andrey Rakhmatullin 4b5fb9b5a6 Add a test for SIGTERM handling. 2023-09-24 22:52:17 +04:00
Andrey Rakhmatullin d19e315b0b Add an interactive test for `scrapy shell`. 2023-09-24 21:51:19 +04:00
Andrey Rakhmatullin 922ff57384 Improve the backwards compatibility for RetryMiddleware.EXCEPTIONS_TO_RETRY. 2023-09-14 15:51:00 +04:00
Andrey Rakhmatullin 6428356584 Use twisted.python.log instead of failure.printTraceback(). 2023-09-13 18:20:42 +04:00
Andrey Rakhmatullin be0e33af92 Don't rely on unhandled exception logging in the expected to fail scripts. 2023-09-13 16:59:54 +04:00
Andrey Rakhmatullin 3756216339 Rename methods. 2023-09-13 15:47:22 +04:00
Andrey Rakhmatullin 619140717f Fix the new addon test. 2023-09-13 15:33:51 +04:00
Andrey Rakhmatullin 028a56b9a2 Improve and simplify tests. 2023-09-13 14:17:41 +04:00
Andrey Rakhmatullin 62183dc25f Merge branch 'master' into change-init-order 2023-09-12 22:16:21 +04:00
Andrey Rakhmatullin 77f39be407 Add integration tests for modifying settings via args. 2023-09-12 20:16:26 +04:00
Andrey Rakhmatullin e26bf4f918
Pin brotli for PyPy tests (#6045) 2023-09-12 16:18:25 +02:00
Andrey Rakhmatullin bb15c93a2b Add Crawler._load_settings(). 2023-09-12 14:44:16 +04:00
Andrey Rakhmatullin 6629a61dd9 Fix one more testcase. 2023-09-08 20:15:49 +04:00
Andrey Rakhmatullin 036d5836d0 Move more things from Crawler.__init__() to .crawl(). 2023-09-08 19:29:05 +04:00
Andrey Rakhmatullin 97b98bf181 Roll back the update_settings() changes. 2023-09-07 17:02:37 +04:00
Maria Mitropoulou 721df895f9
Fix charset handling in MailSender #5096 (#5118) 2023-09-07 07:47:42 +02:00
Andrey Rakhmatullin b4380995da Merge remote-tracking branch 'origin/master' into change-init-order 2023-09-06 20:34:27 +04:00
Andrey Rakhmatullin b39d2d4353 Add a test. 2023-09-06 19:44:22 +04:00
Andrey Rakhmatullin d3b5c9be97 Deprecate running Crawler.crawl() twice. 2023-09-06 19:29:44 +04:00
Andrey Rakhmatullin 69282c5318 Merge branch 'master' into change-init-order 2023-09-05 19:23:36 +04:00
Víctor Ruiz c1dd5493ac
Fix enabled addons not being logged (#6036) 2023-09-04 08:03:35 +02:00
Mikhail Korobov 0fcb0554d1
Merge pull request #6016 from jxlil/fix/response.json 
Fix: response.json() call makes unnecessary memory allocation
 2023-08-31 14:07:24 +05:00
Georgiy Zatserklianyi 1f03cb1419 Merge remote-tracking branch 'upstream/master' into periodic_log_2 2023-08-21 17:07:18 +03:00
Laerte Pereira df2163ce6a
Remove datetime.utcnow() usage (#6014) 2023-08-21 15:51:49 +02:00
Jalil SA fd4292b722 fix: test_cache_json_response 2023-08-17 10:24:34 -06:00
Laerte Pereira b06936f111 Handle Tuple type on getdictorlist method, bump 3.12 python version 2023-08-14 10:33:48 -03:00
Andrey Rakhmatullin 34d050cfe5
Remove deprecated CrawlerRunner.spiders. (#6010) 2023-08-11 10:41:05 +02:00
Laerte Pereira 4a090d951a
Remove deprecated PythonItemExporter.binary (#6007) 2023-08-10 07:36:42 +02:00
Andrey Rakhmatullin 644a71bfd4 Use ftp:// URLs in FTP tests. 2023-08-07 00:17:52 +04:00
Andrey Rakhmatullin 471281d29e Fixes for scrapy/utils/iterators.py typing. 2023-08-06 23:05:02 +04:00
Andrey Rakhmatullin 518e56046e Check for async callbacks in contracts. 2023-08-06 17:28:34 +04:00
Andrey Rakhmatullin d1f87e4f08 More typing for scrapy/utils/iterators.py. 2023-08-06 00:35:33 +04:00
Georgiy Zatserklianyi e9b088f1fb periodic_log: typing 2023-08-04 12:49:22 +03:00
Laerte Pereira 09c63a178b
Remove the deprecated spider parameter from the engine (#5998) 2023-08-04 07:53:04 +02:00
Laerte Pereira b9c32a0cfd
Remove functions deprecated in 2.6.0 (#5996) 2023-08-03 17:06:55 +02:00
Andrey Rakhmatullin 9f9a2292e0
Deprecate the custom attribute of build_component_list(). (#5993) 2023-08-02 14:21:06 +02:00
Andrey Rakhmatullin 06ebdee35d
Merge pull request #5950 from wRAR/addons 
Add-ons
 2023-08-01 22:18:57 +04:00
Andrey Rakhmatullin e58b8078f0 Improve NotConfigured handling when clspath is a class. 2023-08-01 20:20:15 +04:00
Andrey Rakhmatullin 41a4a163e3 Handle NotConfigured in add-ons. 2023-07-31 21:09:18 +04:00
Andrey Rakhmatullin d8c5b41559 Add more addon tests. 2023-07-31 19:09:21 +04:00
Andrey Rakhmatullin bddbbc522a
Merge pull request #5979 from BurnzZ/closespider-timeout-no-item 2023-07-28 12:56:44 +04:00
freddiewanah c4f0aa4fdf
Refactor test cases to improve unit test quality (#5986) 2023-07-26 13:09:03 +02:00
Andrey Rakhmatullin 6e3e3c2172
Merge pull request #5977 from scrapy/typing-settings 
Typing for scrapy.settings.BaseSettings
 2023-07-24 21:40:52 +04:00
Kevin Lloyd Bernal 9a1bf40c2f expose AWS_REGION_NAME in S3FeedStorage 2023-07-24 16:37:52 +08:00
Laerte Pereira 7522aeed35
fix: -O/-o option does not work with absolute paths on Windows (#5971) 2023-07-24 09:53:41 +02:00
Mikhail Korobov af2aa4b421
Merge pull request #5847 from namelessGonbai/feat/FixFeedExport 
Fix FeedExporter not to export empty file
 2023-07-22 23:18:36 +05:00
Adnan Awan 5d91ea12d6
Fix JsonItemExporter puts lone comma in the output if encoder fails (#5952) 
* Partial fix for #3090 - only addresses JSON feeds.

* Adding test case for #3090 to Json Exporter

* Changing the deliberate-fail JSON example to a complex

* Further tightening JsonItemExporter behaviour to prevent corruption.

Based on Mikhail's observation that to_bytes can fail also, leading
to the same dangling comma as the failure to encode to JSON.

Added a new test case to avoid reversion.

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add initial changes from cathal's PR
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Handle exception not to add empty item.
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add comment for handling the exception
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Remove unused import
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Fix invalid json issue
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Perform CR changes
- https://github.com/scrapy/scrapy/issues/3090

---------

Co-authored-by: Cathal Garvey <cathalgarvey@cathalgarvey.me>
 2023-07-22 23:13:40 +05:00
Andrey Rakhmatullin e058a05763 Skip tests that don't work on Python 3.12. 2023-07-22 17:51:13 +04:00
Andrey Rakhmatullin 583df9f7d0 Simplify skipping uvloop tests. 2023-07-22 17:44:37 +04:00
Andrey Rakhmatullin 5c34f34ecb Make AddonManager.add() private. 2023-07-18 19:50:08 +04:00
Kevin Lloyd Bernal 368ab29ffc improve tests by having SlowSpider 2023-07-18 22:29:15 +08:00
Kevin Lloyd Bernal 3f5bbe3a8f introduce CLOSESPIDER_TIMEOUT_NO_ITEM in CloseSpider 2023-07-18 18:30:21 +08:00
Andrey Rakhmatullin 043a24410b Disable pylint for broken code. 2023-07-17 00:13:49 +04:00
Andrey Rakhmatullin 187e8f9a2d Typing for scrapy/settings/__init__.py. 2023-07-17 00:01:36 +04:00
Andrey Rakhmatullin c92c9af075 Add create_instance support to addons. 2023-06-29 15:04:46 +04:00
Georgiy Zatserklianyi 6fd94fdcb3 periodic_log: tests updated (errors fixed) 2023-06-28 19:55:33 +03:00
Andrey Rakhmatullin 9612ae3e93 Remove more code. 2023-06-28 18:44:30 +04:00
Laerte Pereira b6196309cb
fix: Return value instead of `SettingsAttribute` object when using `pop` method (#5963) 2023-06-28 08:28:49 +02:00
Georgiy Zatserklianyi 56c38231b4 periodic_log: tests for logging stats added 2023-06-27 11:48:51 +03:00
Georgiy Zatserklianyi ebce5b4bcb periodic_log: tests for logging deltas added 2023-06-27 09:22:50 +03:00
Andrey Rakhmatullin e7124447f7 Remove unneeded code. 2023-06-26 16:57:46 +04:00
Andrey Rakhmatullin 4e21e3e59d Merge remote-tracking branch 'origin/master' into addons 2023-06-26 14:25:35 +04:00
Georgiy Zatserklianyi 2ce4856508 periodic_log: Exception handling on init updated 2023-06-25 12:16:59 +03:00
Andrey Rakhmatullin 712ee98848 Merge branch 'master' into typing-utils 2023-06-24 18:31:56 +04:00
Laerte Pereira a3f8912d69 chore: Rename test 2023-06-23 00:15:32 -03:00
Laerte Pereira 876feaf339 chore: Use dunder to delete item instead of del keyword to handle immutable settings 2023-06-23 00:14:31 -03:00
Laerte Pereira 080b9bd0b8 chore: Implement `pop` method on `BaseSettings` class 2023-06-22 23:58:03 -03:00
Alex 04ee3303e4
Adding support for Windows of absolute pathlib.Path objects in FeedExporter (#5939) 2023-06-22 07:04:06 +02:00
Andrey Rakhmatullin 1b78e48944
Merge pull request #5146 from elacuesta/case-insensitive-dict 
CaseInsensitiveDict (deprecate CaselessDict)
 2023-06-21 23:48:40 +04:00
Andrey Rakhmatullin 39282d7bd0 Merge branch 'master' into case-insensitive-dict 2023-06-21 22:12:44 +04:00
Marc Hernández 5360ba34bc
IOError and other cleanup (#4716) 2023-06-21 20:08:53 +02:00
Laerte Pereira a93a63c208 fix: move import to inside function 2023-06-20 17:05:49 -03:00
Laerte Pereira 6afb31b82b chore: Add `packaging` to tests deps 2023-06-20 16:48:07 -03:00
Laerte Pereira 0097b4c0bb cleanup: Remove `pkg_resources` usage 2023-06-20 16:40:38 -03:00
Serhii A 777a6ea412
Make the retry middleware exception list configurable (#5929) 2023-06-16 15:46:06 +02:00
Georgiy Zatserklianyi 6e65eeb07b periodic_log: tests [wip] added 2023-06-16 14:30:50 +03:00
Andrey Rakhmatullin 79bf8b1f2e Test cleanup. 2023-06-15 19:28:14 +04:00
Andrey Rakhmatullin 2f9ebb66c3 Remove some dead code/docs. 2023-06-15 17:00:38 +04:00
Andrey Rakhmatullin 58300e066f
Merge pull request #5949 from wRAR/tests-as-installed 
Allow running tests against the installed version
 2023-06-14 23:15:04 +04:00
Andrey Rakhmatullin c7f78a8305 Revert "Remove load_module_or_object." 
This reverts commit 815af43120.
 2023-06-14 20:43:18 +04:00
Andrey Rakhmatullin 7cfdca8f9b Actually run test_set_asyncio_event_loop(). 2023-06-14 18:23:29 +04:00
Andrey Rakhmatullin 815af43120 Remove load_module_or_object. 2023-06-14 18:11:14 +04:00
Andrey Rakhmatullin 7ebb8256f0 Some cleanup. 2023-06-14 17:52:11 +04:00
Andrey Rakhmatullin fdbc141b23 Replace pkg_resources with packaging. 2023-06-14 17:23:50 +04:00
Andrey Rakhmatullin 55ac26228b Remove builtin addons. 2023-06-14 16:50:43 +04:00
Andrey Rakhmatullin 075ad6f196 Test, linter etc. fixes. 2023-06-14 16:34:18 +04:00
Andrey Rakhmatullin 898e3045a1 Merge branch 'master' into addons 2023-06-14 15:37:08 +04:00
Andrey Rakhmatullin 85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export 
Fix #960: S3 Feed Export throws boto error
 2023-06-13 18:44:06 +04:00
Andrey Rakhmatullin 8a64f3e8de
Merge branch 'master' into feat/FixFeedExport 2023-06-13 18:21:06 +04:00
Andrey Rakhmatullin 9e0bfc4a3d
Merge pull request #5705 from srki24/issue5043-feed_export 
S3 CsvItemExporter read of closed file error #5043
 2023-06-13 18:14:27 +04:00
Andrey Rakhmatullin 493ea43538 Improve finding tests.test_cmdline.settings. 2023-06-12 21:50:29 +04:00
Andrey Rakhmatullin 49839d6071 Don't rely on get_testenv() for running mockserver. 2023-06-04 19:59:58 +04:00
Andrey Rakhmatullin 54fa04aa0a Add typing to scrapy/utils/test.py, fix a FTP test. 2023-05-07 22:39:27 +04:00
Andrey Rakhmatullin ea15ff1d32
Merge pull request #5915 from jxlil/feature/support-parsel-jmespath 
Added support for the Parsel JMESPath feature
 2023-05-03 13:18:39 +04:00
Jalil SA a038faf11c fix: tests/tes_selector.py 2023-05-02 19:40:04 -06:00
Jalil SA 4bb99fd2f3 fix: pylint 2023-05-02 19:26:20 -06:00
Jalil SA a604dfae5c update tests 2023-05-02 19:19:00 -06:00
Laerte Pereira 7b49aa1b01 chore: add tests 2023-05-02 12:53:04 -03:00
Jalil SA 578606779d update tests 2023-04-29 00:52:39 -06:00
Jalil SA 3d29f20fc2 added tests for jmespath 2023-04-28 23:54:09 -06:00
guillermo-bondonno b50c032ee9
Add feed_slot_closed and feed_exporter_closed signals (#5876) 2023-04-26 08:20:37 +02:00
Andrey Rakhmatullin abc9c1ac6f
Merge pull request #5901 from jxlil/fix/5899 
Fix: Request.from_curl() with $-prefixed string literals
 2023-04-19 13:35:01 +04:00
tstauder ef61fb5698
Fix flaky tests involving feed export batches (#5898) 
Co-authored-by: jmannoop <jmannoop@umich.edu>
 2023-04-19 08:33:32 +02:00
Jalil SA 88c58a8c9c feat: added test_post_data_raw_with_string_prefix 2023-04-18 22:28:16 -05:00
guillermo-bondonno 441ac196e4
Implement a request_to_curl function (#5892) 2023-04-13 17:46:59 +02:00
Serhii A c2a31974ff
Improve get_func_args (#5885) 2023-04-13 11:44:20 +02:00
Andrey Rakhmatullin d911837389
Merge pull request #5881 from sbartlett97/LinkExtractor-Patch 
Patched LxmlParserLinkExtractor
 2023-04-12 12:28:21 +04:00
Adrián Chaves 3f0c2fae5e Skip test_skip_bad_links based on the w3lib version 2023-04-12 09:28:28 +02:00
Kartik Kumar d47c732ae9
base64-decode GCS checksums (#5891) 2023-04-11 18:25:42 +02:00
Andrey Rakhmatullin 1e20ba0a1b
Merge branch 'master' into feat/FixFeedExport 2023-04-11 19:48:19 +04:00
Felipe Boff Nunes a0e2e36b52 adjustments 2023-04-06 14:23:19 -03:00
Felipe Boff Nunes 0ffd1667ba adjustments 2023-04-06 14:17:35 -03:00
Samuel Bartlett 618e82dbe1 Exclude test for python versionbs less than 3.8 2023-03-31 15:12:47 +00:00
Samuel Bartlett 608b7de582 Skip new test if python version less than 3.8 2023-03-31 14:38:06 +00:00
Samuel Bartlett 4043560547 remove utf-8 encoding flag from test 2023-03-31 12:29:22 +00:00
Samuel Bartlett 00d93026c8 Fix bad test case 2023-03-31 08:30:19 +00:00
Samuel Bartlett 7cb7cf1ad1 Add link extractor back to test 2023-03-31 08:09:02 +00:00
Samuel Bartlett 9cbcf7724d Add test to make sure spider doesn't crash on bad 2023-03-31 08:07:43 +00:00
jazzthief 4ebc08ef10
Switch from `is_boto3_available()` to `IS_BOTO3_AVAILABLE` var 2023-03-16 17:24:11 +01:00
kenshi kikuchi c8ed793257 Fix test_export_no_items_multiple_feeds 2023-03-16 17:16:14 +09:00
Adrián Chaves 7e7b41c6b3 Fix test expectations for boto3 2023-03-15 16:38:13 +01:00
Adrián Chaves 39dbfa1d82 Minimize test reliance on S3; do not install botocore on the default test environments 2023-03-15 15:31:39 +01:00
auxsvr d60b4edd11
Prevent an edge case that creates an extra event loop (#5832) 2023-03-14 09:31:13 +01:00
Alex 101a0c32d7
Support genspider with HTTPS (#5808) 2023-03-14 08:13:44 +01:00
Mohammadtaher Abbasi afafc2781a
Fix parse command issues with asyncio (#5824) 2023-03-10 16:20:15 +01:00
Andrey Rakhmatullin 3659a8c8d9
Merge pull request #5328 from GeorgeA92/per_slot_settings 
Per slot settings
 2023-03-09 22:31:24 +05:00
kenshi kikuchi d311779887 Fix FeedExporter 
+ Fix FeedExporter not to export empty file
+ Change default value of FEED_STORE_EMPTY
 2023-03-08 16:24:09 +09:00
Jalil SA 8aca47e25d
Add setdefault to BaseSettings (#5821) 2023-03-07 10:26:59 +01:00
Alex 78269f9a4f
Merge branch 'scrapy:master' into path_object_error_#5739 2023-02-25 20:49:04 -08:00
Georgiy Zatserklianyi 864eee66c7 per_slot_settings(docs): code style fix 3 (from previous pre-commit check) 2023-02-16 18:19:36 +02:00
Georgiy Zatserklianyi dd5524eb98 per_slot_settings(docs): code style fix 2 (from previous pre-commit check) 2023-02-16 17:59:36 +02:00
Georgiy Zatserklianyi 045092e8d7 per_slot_settings(docs): code style fix (from previous pre-commit check) 2023-02-16 17:55:59 +02:00
Andrey Rakhmatullin 824641e349 Merge remote-tracking branch 'origin/master' into mypy-imports 2023-02-11 22:02:19 +04:00
Alex f17d2e711d
Merge branch 'scrapy:master' into path_object_error_#5739 2023-02-10 00:56:51 -08:00
Georgiy Zatserklianyi 80a86de507 per_slot_settings: test delays increased 2023-02-08 22:40:22 +02:00
Georgiy Zatserklianyi 13d1f69c78 Merge remote-tracking branch 'upstream/master' into per_slot_settings 
# Conflicts:
#	scrapy/core/downloader/__init__.py
 2023-02-08 22:23:13 +02:00
Alex 4692e0e16b Merge branch 'master' into path_object_error_#5739 2023-02-02 11:10:01 -08:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Adrián Chaves 426f3ebb7b Fix typo: causes it be closed → causes it to be closed 2023-02-02 05:58:32 +01:00
Adrián Chaves 32bc8bd436 Merge remote-tracking branch 'scrapy/master' into issue5043-feed_export 2023-02-02 05:56:48 +01:00
Adrián Chaves 2f2bcb006d Test stream detaching in CsvItemExporter 2023-02-02 05:55:59 +01:00
Alex 8c8894f4be
Apply black formatting to templates (#5814) 2023-02-02 04:42:47 +01:00
Andrey Rakhmatullin ccd1385e11
Merge branch 'master' into no-callback 2023-01-30 15:43:05 +05:00
Andrey Rakhmatullin da15d93d39
Merge pull request #5806 from jxlil/master 
Use isort
 2023-01-30 15:40:44 +05:00
Andrey Rakhmatullin c0efb271a2 Ignore typing for twisted.internet.reactor globally. 2023-01-27 23:19:44 +04:00
Adrián Chaves f03b47db05 Make NO_CALLBACK a callable 2023-01-27 17:35:32 +01:00
Alex a1e2fbafdc applied black to tests 2023-01-26 16:46:08 -08:00
Cj Malone 33b85a9e2a
Test local files aren't processed 2023-01-26 19:51:20 +00:00
Alex 5433015a25
Merge branch 'master' into path_object_error_#5739 2023-01-26 07:12:01 -08:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Andrey Rakhmatullin afd5d85320 Fix remaining issues in tests. 2023-01-25 22:15:10 +04:00
Andrey Rakhmatullin 2e33fb812b Silence improper typing of twisted.internet.reactor in tests. 2023-01-25 22:15:07 +04:00
Andrey Rakhmatullin c0ea7fd4fd Remove obsolete top-level run code in tests. 2023-01-25 22:14:24 +04:00
Adrián Chaves 9272c4af0c Merge remote-tracking branch 'scrapy/master' into no-callback 2023-01-25 17:41:41 +01:00
Alex 085b340e8a
Merge branch 'scrapy:master' into path_object_error_#5739 2023-01-22 21:46:28 -08:00
Emmanuel Rondan f2c22aaabb Merge remote-tracking branch 'upstream/master' into add-black-formatter 2023-01-20 17:08:07 -03:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Alex b1dd893fbb Support Path Objects Issue #5739 2023-01-20 02:17:02 -08:00
Tobias Mayr f449ee5377
Fix SMTP STARTTLS for Twisted >= 21.2.0 (#5406) 2023-01-19 19:44:55 +01:00
Adrián Chaves 50500a6b28 Implement a NO_CALLBACK value for Request.callback 2023-01-19 17:14:18 +01:00
Andrey Rakhmatullin 93ad6a4bc2 Simplify code for modern pyOpenSSL. 2023-01-13 20:39:44 +04:00
Andrey Rakhmatullin 74b8c34de6
Merge pull request #5458 from reidemeister94/fix/unique-list-link-extractors 
fix: return unique_list only when link_extractor.unique is True
 2023-01-13 01:13:45 +05:00
Andrey Rakhmatullin 604ddf82fc
Merge pull request #5758 from MattyMay/finish_exporting_for_each_start_exporting 
Finish exporting for each start exporting
 2023-01-13 00:17:17 +05:00
Andrey Rakhmatullin 2944894c4d
Merge pull request #5764 from jsoref/spelling 
Spelling
 2023-01-13 00:15:36 +05:00
spav b386c64864 Fix tests for allowing duplicates in extract_links 2023-01-11 19:58:42 +01:00
Adrián Chaves 520be4ff0b Merge remote-tracking branch 'scrapy/master' into fix/http2_cookies 2023-01-11 15:59:59 +01:00
Adrián Chaves d9f6de6bf5 Add a test for receiving duplicate headers 2023-01-11 15:59:45 +01:00
Adrián Chaves 1ab900659e
Fix typo: finis → finish 2023-01-11 14:05:13 +01:00
Adrián Chaves 724b033287
Fix CI issues related to asyncio (#5782) 2023-01-06 19:51:07 +01:00
Adrián Chaves fd6742e811
Fix tests for pygments 2.14 (#5783) 2023-01-02 19:13:31 +01:00
Emmanuel Rondan c5cdd0d30c reaplying black again and removing line length on pre-commit black config 2022-12-29 12:48:20 -03:00
Emmanuel Rondan 303f0a70fc re applying black to the code with default line length 2022-12-29 11:26:06 -03:00
Emmanuel Rondan 12b556a352
Merge branch 'master' into add-black-formatter 2022-12-26 12:25:32 -03:00
Matt Mayfield a0832081ba Merge branch 'master' into finish_exporting_for_each_start_exporting 2022-12-19 18:12:02 -05:00
Matt Mayfield 0a84ce448c Fix InstrumentedFeedSlot 
I accidentally called the wrong super method in overriden
finish_exporting
 2022-12-19 18:09:43 -05:00
Josh Soref 8a526d161c spelling: user 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:46:11 -05:00
Josh Soref 860fbef608 spelling: unknown 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 1e44d4614e spelling: straight 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref d27c611cc0 spelling: referrer 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 6aa5374bd3 spelling: received 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 826e0ee611 spelling: preexisting 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 66ab82f126 spelling: precedence 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 8ae77fdb34 spelling: pipeline 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 1a5cf00db7 spelling: overridden 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref a839b61147 spelling: nonexistent 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref e894db2f3f spelling: laziness 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref a26b6b0607 spelling: indistinct 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 1300c1c881 spelling: children 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Alex 93ace91ace
Merge branch 'scrapy:master' into fix_shell_fetch_alt 2022-12-13 21:50:28 -08:00
Alex c2de9372a2 added unittest for scrapy shell command and fixed flake8 error 2022-12-13 21:37:17 -08:00
Matt Mayfield 40f4b262d2 Fix style errors 2022-12-12 12:36:29 -05:00
Matt Mayfield 8d67a08155 Change test name and add additional tests 2022-12-12 11:55:42 -05:00
Matt Mayfield 66f127eb37 Make test cleaner and more reusable 2022-12-12 11:46:05 -05:00
Laerte Pereira e099572cec
change xfail condition 2022-12-12 09:56:50 -03:00
Laerte Pereira 0dbd1d9b81
use xfail, switch to packaging to parse version 2022-12-12 08:58:40 -03:00
Matt Mayfield 087334009c Call `finish_exporting` even when itemcount == 0 2022-12-11 23:12:41 -05:00
Laerte Pereira 6757973b61
fix test_follow_whitespace_* tests 2022-12-11 09:11:55 -03:00
Emmanuel Rondan 1a3db81492 ignoring E203 flake8 error 
ignoring implicit-str-concat pylint error
fixing pylint comment on test_request_cb_kwargs.py
 2022-11-29 15:06:54 -03:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Andrey Rakhmatullin f85c3f3d68 Add a comment about `tmpname + '^'`. 2022-11-25 18:46:14 +06:00
Laerte Pereira e769532644
Remove `noconnect` deprecate code 2022-11-23 08:30:11 -03:00
Laerte Pereira 1506479672
Remove deprecated test 2022-11-22 10:07:32 -03:00
Laerte Pereira 8e0025f53d
Remove support for override settings with `SCRAPY_` environment variables 2022-11-22 09:38:54 -03:00
Laerte Pereira 1a6408c3fa
Remove `FilteringLinkExtractor` 2022-11-21 07:30:20 -03:00
Laerte Pereira d5b6c236a9
Remove deprecated code (#5719) 2022-11-21 09:32:26 +01:00
GeorgeA92 042012f6bd per slot settings: error calculation metho updated 2022-11-18 14:56:29 +02:00
Andrey Rahmatullin bbe24d79a5
Fix test issues 2022-11-07 17:08:54 +05:00
Andrey Rahmatullin 6c0890ff54
Simplify the changes after the merge 2022-11-07 16:36:54 +05:00
Andrey Rahmatullin b03d84e9fb
Merge branch 'master' into 3689-update 2022-11-07 16:33:21 +05:00
Andrey Rahmatullin 77cd511a5e
Merge pull request #5489 from PluT00/deprecate-NoimagesDrop 
Deprecate scrapy.pipelines.images.NoimagesDrop
 2022-11-05 14:18:19 +05:00
Andrey Rahmatullin 8004075823
Merge pull request #5699 from elacuesta/crawlspider-callback-keyword-arguments 
CrawlSpider: pass cb_kwargs from process_request
 2022-11-02 17:35:03 +05:00
Andrey Rahmatullin a95a338eea
Merge pull request #5599 from tonal/patch-1 
Partial in is_generator_with_return_value
 2022-10-31 20:08:16 +05:00
Eugenio Lacuesta b18560315b
Add tests 2022-10-30 18:28:16 -03:00
Andrey Rakhmatullin b71d0292d5 Add a test for processing partial callbacks. 2022-10-27 18:13:47 +06:00
Godson 3a34fa8399
Get the event loop from event_loop_policy to avoid a deprecation warning (#5689) 2022-10-27 14:02:12 +02:00
Andrey Rakhmatullin ca50af6453 Remove an unused import. 2022-10-27 17:12:26 +06:00
Andrey Rahmatullin 5780ccba55
Merge branch 'master' into deprecate-NoimagesDrop 2022-10-27 15:46:32 +05:00
Adrián Chaves 46dd152b3e Merge remote-tracking branch 'upstream/master' into more-lenient-proxying 2022-10-26 16:55:04 +02:00
Godson-Gnanaraj 830e1c5dd8 Add test for parsing decorated methods 2022-10-26 01:26:54 +05:30
Andrey Rahmatullin b33244e2f0
Fix the flake8 per-file ignore syntax (#5688) 2022-10-21 16:17:04 +02:00
Matthew Donoughe 12a26755ae
improve render_templatefile 2022-10-20 20:14:43 -04:00
Matthew Donoughe c85de90819
fix path to tox.ini 2022-10-20 19:44:35 -04:00
Matthew Donoughe 065db7b566
fix some mypy issues 2022-10-19 08:28:26 -04:00
Matthew Donoughe 93d82648e5
resolve this path too 2022-10-19 08:08:21 -04:00
Matthew Donoughe fb26e6b650
resolve path to search in error 2022-10-18 20:21:11 -04:00
Matthew Donoughe 7daf735f45
downgrade type hint syntax 2022-10-18 20:07:29 -04:00