RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,323 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

2233 Commits

Author SHA1 Message Date
Georgiy Zatserklianyi 6fd94fdcb3 periodic_log: tests updated (errors fixed) 2023-06-28 19:55:33 +03:00
Andrey Rakhmatullin 9612ae3e93 Remove more code. 2023-06-28 18:44:30 +04:00
Laerte Pereira b6196309cb
fix: Return value instead of `SettingsAttribute` object when using `pop` method (#5963) 2023-06-28 08:28:49 +02:00
Georgiy Zatserklianyi 56c38231b4 periodic_log: tests for logging stats added 2023-06-27 11:48:51 +03:00
Georgiy Zatserklianyi ebce5b4bcb periodic_log: tests for logging deltas added 2023-06-27 09:22:50 +03:00
Andrey Rakhmatullin e7124447f7 Remove unneeded code. 2023-06-26 16:57:46 +04:00
Andrey Rakhmatullin 4e21e3e59d Merge remote-tracking branch 'origin/master' into addons 2023-06-26 14:25:35 +04:00
Georgiy Zatserklianyi 2ce4856508 periodic_log: Exception handling on init updated 2023-06-25 12:16:59 +03:00
Andrey Rakhmatullin 712ee98848 Merge branch 'master' into typing-utils 2023-06-24 18:31:56 +04:00
Laerte Pereira a3f8912d69 chore: Rename test 2023-06-23 00:15:32 -03:00
Laerte Pereira 876feaf339 chore: Use dunder to delete item instead of del keyword to handle immutable settings 2023-06-23 00:14:31 -03:00
Laerte Pereira 080b9bd0b8 chore: Implement `pop` method on `BaseSettings` class 2023-06-22 23:58:03 -03:00
Alex 04ee3303e4
Adding support for Windows of absolute pathlib.Path objects in FeedExporter (#5939) 2023-06-22 07:04:06 +02:00
Andrey Rakhmatullin 1b78e48944
Merge pull request #5146 from elacuesta/case-insensitive-dict 
CaseInsensitiveDict (deprecate CaselessDict)
 2023-06-21 23:48:40 +04:00
Andrey Rakhmatullin 39282d7bd0 Merge branch 'master' into case-insensitive-dict 2023-06-21 22:12:44 +04:00
Marc Hernández 5360ba34bc
IOError and other cleanup (#4716) 2023-06-21 20:08:53 +02:00
Laerte Pereira a93a63c208 fix: move import to inside function 2023-06-20 17:05:49 -03:00
Laerte Pereira 6afb31b82b chore: Add `packaging` to tests deps 2023-06-20 16:48:07 -03:00
Laerte Pereira 0097b4c0bb cleanup: Remove `pkg_resources` usage 2023-06-20 16:40:38 -03:00
Serhii A 777a6ea412
Make the retry middleware exception list configurable (#5929) 2023-06-16 15:46:06 +02:00
Georgiy Zatserklianyi 6e65eeb07b periodic_log: tests [wip] added 2023-06-16 14:30:50 +03:00
Andrey Rakhmatullin 79bf8b1f2e Test cleanup. 2023-06-15 19:28:14 +04:00
Andrey Rakhmatullin 2f9ebb66c3 Remove some dead code/docs. 2023-06-15 17:00:38 +04:00
Andrey Rakhmatullin 58300e066f
Merge pull request #5949 from wRAR/tests-as-installed 
Allow running tests against the installed version
 2023-06-14 23:15:04 +04:00
Andrey Rakhmatullin c7f78a8305 Revert "Remove load_module_or_object." 
This reverts commit 815af43120.
 2023-06-14 20:43:18 +04:00
Andrey Rakhmatullin 7cfdca8f9b Actually run test_set_asyncio_event_loop(). 2023-06-14 18:23:29 +04:00
Andrey Rakhmatullin 815af43120 Remove load_module_or_object. 2023-06-14 18:11:14 +04:00
Andrey Rakhmatullin 7ebb8256f0 Some cleanup. 2023-06-14 17:52:11 +04:00
Andrey Rakhmatullin fdbc141b23 Replace pkg_resources with packaging. 2023-06-14 17:23:50 +04:00
Andrey Rakhmatullin 55ac26228b Remove builtin addons. 2023-06-14 16:50:43 +04:00
Andrey Rakhmatullin 075ad6f196 Test, linter etc. fixes. 2023-06-14 16:34:18 +04:00
Andrey Rakhmatullin 898e3045a1 Merge branch 'master' into addons 2023-06-14 15:37:08 +04:00
Andrey Rakhmatullin 85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export 
Fix #960: S3 Feed Export throws boto error
 2023-06-13 18:44:06 +04:00
Andrey Rakhmatullin 8a64f3e8de
Merge branch 'master' into feat/FixFeedExport 2023-06-13 18:21:06 +04:00
Andrey Rakhmatullin 9e0bfc4a3d
Merge pull request #5705 from srki24/issue5043-feed_export 
S3 CsvItemExporter read of closed file error #5043
 2023-06-13 18:14:27 +04:00
Andrey Rakhmatullin 493ea43538 Improve finding tests.test_cmdline.settings. 2023-06-12 21:50:29 +04:00
Andrey Rakhmatullin 49839d6071 Don't rely on get_testenv() for running mockserver. 2023-06-04 19:59:58 +04:00
Andrey Rakhmatullin 54fa04aa0a Add typing to scrapy/utils/test.py, fix a FTP test. 2023-05-07 22:39:27 +04:00
Andrey Rakhmatullin ea15ff1d32
Merge pull request #5915 from jxlil/feature/support-parsel-jmespath 
Added support for the Parsel JMESPath feature
 2023-05-03 13:18:39 +04:00
Jalil SA a038faf11c fix: tests/tes_selector.py 2023-05-02 19:40:04 -06:00
Jalil SA 4bb99fd2f3 fix: pylint 2023-05-02 19:26:20 -06:00
Jalil SA a604dfae5c update tests 2023-05-02 19:19:00 -06:00
Laerte Pereira 7b49aa1b01 chore: add tests 2023-05-02 12:53:04 -03:00
Jalil SA 578606779d update tests 2023-04-29 00:52:39 -06:00
Jalil SA 3d29f20fc2 added tests for jmespath 2023-04-28 23:54:09 -06:00
guillermo-bondonno b50c032ee9
Add feed_slot_closed and feed_exporter_closed signals (#5876) 2023-04-26 08:20:37 +02:00
Andrey Rakhmatullin abc9c1ac6f
Merge pull request #5901 from jxlil/fix/5899 
Fix: Request.from_curl() with $-prefixed string literals
 2023-04-19 13:35:01 +04:00
tstauder ef61fb5698
Fix flaky tests involving feed export batches (#5898) 
Co-authored-by: jmannoop <jmannoop@umich.edu>
 2023-04-19 08:33:32 +02:00
Jalil SA 88c58a8c9c feat: added test_post_data_raw_with_string_prefix 2023-04-18 22:28:16 -05:00
guillermo-bondonno 441ac196e4
Implement a request_to_curl function (#5892) 2023-04-13 17:46:59 +02:00
Serhii A c2a31974ff
Improve get_func_args (#5885) 2023-04-13 11:44:20 +02:00
Andrey Rakhmatullin d911837389
Merge pull request #5881 from sbartlett97/LinkExtractor-Patch 
Patched LxmlParserLinkExtractor
 2023-04-12 12:28:21 +04:00
Adrián Chaves 3f0c2fae5e Skip test_skip_bad_links based on the w3lib version 2023-04-12 09:28:28 +02:00
Kartik Kumar d47c732ae9
base64-decode GCS checksums (#5891) 2023-04-11 18:25:42 +02:00
Andrey Rakhmatullin 1e20ba0a1b
Merge branch 'master' into feat/FixFeedExport 2023-04-11 19:48:19 +04:00
Felipe Boff Nunes a0e2e36b52 adjustments 2023-04-06 14:23:19 -03:00
Felipe Boff Nunes 0ffd1667ba adjustments 2023-04-06 14:17:35 -03:00
Samuel Bartlett 618e82dbe1 Exclude test for python versionbs less than 3.8 2023-03-31 15:12:47 +00:00
Samuel Bartlett 608b7de582 Skip new test if python version less than 3.8 2023-03-31 14:38:06 +00:00
Samuel Bartlett 4043560547 remove utf-8 encoding flag from test 2023-03-31 12:29:22 +00:00
Samuel Bartlett 00d93026c8 Fix bad test case 2023-03-31 08:30:19 +00:00
Samuel Bartlett 7cb7cf1ad1 Add link extractor back to test 2023-03-31 08:09:02 +00:00
Samuel Bartlett 9cbcf7724d Add test to make sure spider doesn't crash on bad 2023-03-31 08:07:43 +00:00
jazzthief 4ebc08ef10
Switch from `is_boto3_available()` to `IS_BOTO3_AVAILABLE` var 2023-03-16 17:24:11 +01:00
kenshi kikuchi c8ed793257 Fix test_export_no_items_multiple_feeds 2023-03-16 17:16:14 +09:00
Adrián Chaves 7e7b41c6b3 Fix test expectations for boto3 2023-03-15 16:38:13 +01:00
Adrián Chaves 39dbfa1d82 Minimize test reliance on S3; do not install botocore on the default test environments 2023-03-15 15:31:39 +01:00
auxsvr d60b4edd11
Prevent an edge case that creates an extra event loop (#5832) 2023-03-14 09:31:13 +01:00
Alex 101a0c32d7
Support genspider with HTTPS (#5808) 2023-03-14 08:13:44 +01:00
Mohammadtaher Abbasi afafc2781a
Fix parse command issues with asyncio (#5824) 2023-03-10 16:20:15 +01:00
Andrey Rakhmatullin 3659a8c8d9
Merge pull request #5328 from GeorgeA92/per_slot_settings 
Per slot settings
 2023-03-09 22:31:24 +05:00
kenshi kikuchi d311779887 Fix FeedExporter 
+ Fix FeedExporter not to export empty file
+ Change default value of FEED_STORE_EMPTY
 2023-03-08 16:24:09 +09:00
Jalil SA 8aca47e25d
Add setdefault to BaseSettings (#5821) 2023-03-07 10:26:59 +01:00
Alex 78269f9a4f
Merge branch 'scrapy:master' into path_object_error_#5739 2023-02-25 20:49:04 -08:00
Georgiy Zatserklianyi 864eee66c7 per_slot_settings(docs): code style fix 3 (from previous pre-commit check) 2023-02-16 18:19:36 +02:00
Georgiy Zatserklianyi dd5524eb98 per_slot_settings(docs): code style fix 2 (from previous pre-commit check) 2023-02-16 17:59:36 +02:00
Georgiy Zatserklianyi 045092e8d7 per_slot_settings(docs): code style fix (from previous pre-commit check) 2023-02-16 17:55:59 +02:00
Andrey Rakhmatullin 824641e349 Merge remote-tracking branch 'origin/master' into mypy-imports 2023-02-11 22:02:19 +04:00
Alex f17d2e711d
Merge branch 'scrapy:master' into path_object_error_#5739 2023-02-10 00:56:51 -08:00
Georgiy Zatserklianyi 80a86de507 per_slot_settings: test delays increased 2023-02-08 22:40:22 +02:00
Georgiy Zatserklianyi 13d1f69c78 Merge remote-tracking branch 'upstream/master' into per_slot_settings 
# Conflicts:
#	scrapy/core/downloader/__init__.py
 2023-02-08 22:23:13 +02:00
Alex 4692e0e16b Merge branch 'master' into path_object_error_#5739 2023-02-02 11:10:01 -08:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Adrián Chaves 426f3ebb7b Fix typo: causes it be closed → causes it to be closed 2023-02-02 05:58:32 +01:00
Adrián Chaves 32bc8bd436 Merge remote-tracking branch 'scrapy/master' into issue5043-feed_export 2023-02-02 05:56:48 +01:00
Adrián Chaves 2f2bcb006d Test stream detaching in CsvItemExporter 2023-02-02 05:55:59 +01:00
Alex 8c8894f4be
Apply black formatting to templates (#5814) 2023-02-02 04:42:47 +01:00
Andrey Rakhmatullin ccd1385e11
Merge branch 'master' into no-callback 2023-01-30 15:43:05 +05:00
Andrey Rakhmatullin da15d93d39
Merge pull request #5806 from jxlil/master 
Use isort
 2023-01-30 15:40:44 +05:00
Andrey Rakhmatullin c0efb271a2 Ignore typing for twisted.internet.reactor globally. 2023-01-27 23:19:44 +04:00
Adrián Chaves f03b47db05 Make NO_CALLBACK a callable 2023-01-27 17:35:32 +01:00
Alex a1e2fbafdc applied black to tests 2023-01-26 16:46:08 -08:00
Cj Malone 33b85a9e2a
Test local files aren't processed 2023-01-26 19:51:20 +00:00
Alex 5433015a25
Merge branch 'master' into path_object_error_#5739 2023-01-26 07:12:01 -08:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Andrey Rakhmatullin afd5d85320 Fix remaining issues in tests. 2023-01-25 22:15:10 +04:00
Andrey Rakhmatullin 2e33fb812b Silence improper typing of twisted.internet.reactor in tests. 2023-01-25 22:15:07 +04:00
Andrey Rakhmatullin c0ea7fd4fd Remove obsolete top-level run code in tests. 2023-01-25 22:14:24 +04:00
Adrián Chaves 9272c4af0c Merge remote-tracking branch 'scrapy/master' into no-callback 2023-01-25 17:41:41 +01:00
Alex 085b340e8a
Merge branch 'scrapy:master' into path_object_error_#5739 2023-01-22 21:46:28 -08:00
Emmanuel Rondan f2c22aaabb Merge remote-tracking branch 'upstream/master' into add-black-formatter 2023-01-20 17:08:07 -03:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Alex b1dd893fbb Support Path Objects Issue #5739 2023-01-20 02:17:02 -08:00
Tobias Mayr f449ee5377
Fix SMTP STARTTLS for Twisted >= 21.2.0 (#5406) 2023-01-19 19:44:55 +01:00
Adrián Chaves 50500a6b28 Implement a NO_CALLBACK value for Request.callback 2023-01-19 17:14:18 +01:00
Andrey Rakhmatullin 93ad6a4bc2 Simplify code for modern pyOpenSSL. 2023-01-13 20:39:44 +04:00
Andrey Rakhmatullin 74b8c34de6
Merge pull request #5458 from reidemeister94/fix/unique-list-link-extractors 
fix: return unique_list only when link_extractor.unique is True
 2023-01-13 01:13:45 +05:00
Andrey Rakhmatullin 604ddf82fc
Merge pull request #5758 from MattyMay/finish_exporting_for_each_start_exporting 
Finish exporting for each start exporting
 2023-01-13 00:17:17 +05:00
Andrey Rakhmatullin 2944894c4d
Merge pull request #5764 from jsoref/spelling 
Spelling
 2023-01-13 00:15:36 +05:00
spav b386c64864 Fix tests for allowing duplicates in extract_links 2023-01-11 19:58:42 +01:00
Adrián Chaves 520be4ff0b Merge remote-tracking branch 'scrapy/master' into fix/http2_cookies 2023-01-11 15:59:59 +01:00
Adrián Chaves d9f6de6bf5 Add a test for receiving duplicate headers 2023-01-11 15:59:45 +01:00
Adrián Chaves 1ab900659e
Fix typo: finis → finish 2023-01-11 14:05:13 +01:00
Adrián Chaves 1ade9d2ed7 Minor test changes 2023-01-10 21:41:49 +01:00
Adrián Chaves 7bb5289ced Cover additional behavior changes in tests 2023-01-10 21:11:57 +01:00
Adrián Chaves 86baef02a0 Test Content-Encoding against Content-Disposition, URL and body 2023-01-10 20:51:43 +01:00
Adrián Chaves deab93b243 Remove duplicate test 2023-01-10 19:04:52 +01:00
Adrián Chaves fc2833d6bd Fix _get_encoding_or_mime_type_from_headers handling on comma-separated Content-Encoding values 2023-01-10 18:32:27 +01:00
Adrián Chaves ee6f5c56b2 Support multi-encoding Content-Encoding headers 2023-01-10 17:55:35 +01:00
Adrián Chaves 68fccc9f04 Test that Content-Disposition takes precedence over body 2023-01-10 16:45:33 +01:00
Adrián Chaves f538252d3b crazy_case → odd_capitalize 2023-01-10 16:40:21 +01:00
Adrián Chaves 14464388e8 Address flake8 issues 2023-01-10 16:38:42 +01:00
Adrián Chaves dd4f4f094e Test handling of feeds mislabeled as HTML 2023-01-10 16:33:54 +01:00
Adrián Chaves 2178521a79 Make sure that supplied “unknown” MIME types from Content-Type are ignored 2023-01-10 13:41:36 +01:00
Adrián Chaves 8908151608 Test that MIME parameters do not break response class choice 2023-01-10 13:01:25 +01:00
Adrián Chaves 45b95c63d7 Address issues reported by pylint 2023-01-10 11:30:06 +01:00
Adrián Chaves e7e201d0b9 Extend tests covering body-based response class choice 2023-01-10 11:15:56 +01:00
Adrián Chaves eef1f127de test_response_class_choosing_request: update after recent changes 2023-01-10 07:43:14 +01:00
Adrián Chaves fcc9e90553 Wrap XHTML with XmlResponse 2023-01-10 07:30:40 +01:00
Adrián Chaves 827de3d288 Finish cleaning up POST_XTRACTMIME_SCENARIOS 2023-01-08 17:22:12 +01:00
Adrián Chaves 711a6093e1 Finish cleaning up PRE_XTRACTMIME_SCENARIOS 2023-01-08 17:02:56 +01:00
Adrián Chaves b2f2788bd4 Add tests for binary-vs-text body 2023-01-08 16:36:35 +01:00
Adrián Chaves f9f9d61457 Always prefer Content-Type over Content-Disposition 2023-01-08 15:46:32 +01:00
Adrián Chaves e7d36a305d Ignore file path extensions in HTTP responses 2023-01-08 14:24:54 +01:00
Adrián Chaves fa3ba512d9 Uncomment test scenarios 2023-01-08 13:21:36 +01:00
Adrián Chaves f029be3aa7 Fix Apache bug handling 2023-01-08 12:55:17 +01:00
Adrián Chaves 160096b7cf pytest-cov: 3.0.0 → 4.0.0 
Removes 2 warnings when running tests
 2023-01-08 11:43:52 +01:00
Adrián Chaves e96b07bd4b flake8: ignore failing file for the time being 2023-01-08 11:38:46 +01:00
Adrián Chaves c6d416c69f Merge remote-tracking branch 'upstream/master' into integrate-mime 2023-01-08 11:25:00 +01:00
Adrián Chaves 724b033287
Fix CI issues related to asyncio (#5782) 2023-01-06 19:51:07 +01:00
Adrián Chaves fd6742e811
Fix tests for pygments 2.14 (#5783) 2023-01-02 19:13:31 +01:00
Adrián Chaves b336980bd1 Merge remote-tracking branch 'upstream/master' into integrate-mime 2023-01-02 13:14:39 +01:00
Emmanuel Rondan c5cdd0d30c reaplying black again and removing line length on pre-commit black config 2022-12-29 12:48:20 -03:00
Emmanuel Rondan 303f0a70fc re applying black to the code with default line length 2022-12-29 11:26:06 -03:00
Emmanuel Rondan 12b556a352
Merge branch 'master' into add-black-formatter 2022-12-26 12:25:32 -03:00
Matt Mayfield a0832081ba Merge branch 'master' into finish_exporting_for_each_start_exporting 2022-12-19 18:12:02 -05:00
Matt Mayfield 0a84ce448c Fix InstrumentedFeedSlot 
I accidentally called the wrong super method in overriden
finish_exporting
 2022-12-19 18:09:43 -05:00
Josh Soref 8a526d161c spelling: user 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:46:11 -05:00
Josh Soref 860fbef608 spelling: unknown 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 1e44d4614e spelling: straight 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00