RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,191 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

11191 Commits

Author SHA1 Message Date
Samuel Bartlett 9cbcf7724d Add test to make sure spider doesn't crash on bad 2023-03-31 08:07:43 +00:00
Samuel Bartlett 9ef00c5c0b Add logging Lines 
Adds an error loggign line to the LinkExtractor to detail encountered
bad links
 2023-03-31 08:01:54 +00:00
Samuel Bartlett 90ce6589ee Add try/except to safe_url_string() 
Added a try catch condition to the safe_url_string() processing
in the LxmlParserLinkExtractor class to avoid scrapers crashing
unneccessarily
 2023-03-30 13:07:51 +00:00
Prathm-s b83fa60a0a return added to _sent_failed 2023-03-30 16:57:56 +05:30
Andrey Rakhmatullin 8045d7eaa5
Merge pull request #5851 from felipeboffnunes/extend_contributing 
#5109 extend contributing.rst
 2023-03-29 16:27:41 +04:00
Andrey Rakhmatullin 4249fc64d1
Merge pull request #5879 from xt1033/bugfix/updated-Makefile-to-enable-string-concatenation 
Fixed Docs Makefile to open build/html/index.html in browser(#5878)
 2023-03-29 12:06:07 +04:00
karza_abhishek 46bb7b31d1 Fixed Docs Makefile to open build/html/index.html in browser(#5878) 2023-03-28 23:23:32 +05:30
Andrey Rakhmatullin 4dacad0d8f
Merge pull request #5877 from jxlil/master 
Fix: `Uncaught ReferenceError: analytics is not defined`
 2023-03-28 14:05:47 +04:00
Jalil SA 2c31aa6c85 delete _template/layout.html 2023-03-28 01:12:19 -06:00
Jalil SA c22c7bd82b fix: rollback GA code 2023-03-27 20:41:19 -06:00
Felipe Boff Nunes af730df83c adjustments 2023-03-27 08:40:17 -03:00
Andrey Rakhmatullin ada9173078
Don't call SSL_get_server_tmp_key() if not available (#5858) 2023-03-27 12:44:44 +02:00
Jalil SA 44cdaa442b
Update docs/topics/spiders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-03-17 13:19:03 -06:00
Jalil SA 24f28c415c fix: docs update_settings() 2023-03-17 12:16:08 -06:00
Jalil SA a1fc37cbff
Update docs/topics/spiders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-03-17 12:13:05 -06:00
Jalil SA 495372648c fix: docs update_settings() 2023-03-16 23:14:57 -06:00
jazzthief cb67bc17b7
Remove `botocore` from docs 2023-03-16 17:25:05 +01:00
jazzthief 4ebc08ef10
Switch from `is_boto3_available()` to `IS_BOTO3_AVAILABLE` var 2023-03-16 17:24:11 +01:00
jazzthief a17d996da2
Change `boto3` version to 1.20.0 for `extra-deps-pinned` env 2023-03-16 17:20:22 +01:00
felipeboffnunes 7bcbfabdbc forgot header bind 2023-03-16 10:28:12 -03:00
felipeboffnunes a81fb5002b adjustments, lean approach 2023-03-16 10:23:06 -03:00
felipeboffnunes 3e59b0805e #5109 extend contributing.rst 2023-03-16 10:02:30 -03:00
namelessGonbai 6ab49e954f
Update docs/topics/feed-exports.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-03-16 18:03:06 +09:00
kenshi kikuchi 50801c7207 Fix Docs 2023-03-16 17:17:20 +09:00
kenshi kikuchi c8ed793257 Fix test_export_no_items_multiple_feeds 2023-03-16 17:16:14 +09:00
Yegor Statkevich ffef837a1f
Merge pull request #1 from Gallaecio/issue_960_s3_feed_export 
Adapt tests for a switch to boto3 with botocore support for backward compatibility
 2023-03-15 18:51:37 +03:00
Adrián Chaves 7e7b41c6b3 Fix test expectations for boto3 2023-03-15 16:38:13 +01:00
Adrián Chaves 590955fac8 Provide separate test environments for botocore and boto3 extras; add extra-deps-pinned and remote extras from pinned 2023-03-15 16:03:44 +01:00
Adrián Chaves 39dbfa1d82 Minimize test reliance on S3; do not install botocore on the default test environments 2023-03-15 15:31:39 +01:00
Adrián Chaves dfbb63a2f1 Merge remote-tracking branch 'scrapy/master' into issue_960_s3_feed_export 2023-03-15 14:07:54 +01:00
auxsvr d60b4edd11
Prevent an edge case that creates an extra event loop (#5832) 2023-03-14 09:31:13 +01:00
Alex 101a0c32d7
Support genspider with HTTPS (#5808) 2023-03-14 08:13:44 +01:00
Hugo van Kemenade 9411cf4e70
Replace deprecated BadZipfile with BadZipFile (#5849) 2023-03-13 19:03:41 +01:00
Mohammadtaher Abbasi afafc2781a
Fix parse command issues with asyncio (#5824) 2023-03-10 16:20:15 +01:00
Andrey Rakhmatullin 3659a8c8d9
Merge pull request #5328 from GeorgeA92/per_slot_settings 
Per slot settings
 2023-03-09 22:31:24 +05:00
Jalil SA 96d51c3afa docs: update 2023-03-08 04:21:21 -06:00
Jalil SA 1d862d0831 fix: remove line breaks 2023-03-08 03:26:38 -06:00
Jalil SA 05893e1796 docs: Spider.update_settings 2023-03-08 02:59:47 -06:00
kenshi kikuchi d311779887 Fix FeedExporter 
+ Fix FeedExporter not to export empty file
+ Change default value of FEED_STORE_EMPTY
 2023-03-08 16:24:09 +09:00
Jalil SA 8aca47e25d
Add setdefault to BaseSettings (#5821) 2023-03-07 10:26:59 +01:00
Georgiy Zatserklianyi 218829b1db per slot settings: typo fixed 2023-03-06 16:25:49 +02:00
Georgiy Zatserklianyi be52fe4f67 Revert "periodic log stats: typo in docs fixed" 
This reverts commit 68ba25cb69.
 2023-03-06 16:24:28 +02:00
Georgiy Zatserklianyi 68ba25cb69 periodic log stats: typo in docs fixed 2023-03-04 22:15:06 +02:00
Andrey Rakhmatullin 88eec7a417
Merge pull request #5801 from alexpdev/path_object_error_#5739 
Support Path Objects Issue #5739
 2023-03-04 22:01:24 +05:00
Alex 59f5250e59
Merge branch 'scrapy:master' into path_object_error_#5739 2023-03-03 16:47:08 -08:00
Oleg 2b3a8f0d69
Fix a typo in the LOG_FORMAT description (#5839) 2023-03-02 10:19:41 +01:00
Alex 78269f9a4f
Merge branch 'scrapy:master' into path_object_error_#5739 2023-02-25 20:49:04 -08:00
Andrey Rakhmatullin 8fbebfa943
Merge pull request #5805 from scrapy/mypy-imports 
Handle typing issues hidden by follow_imports=skip
 2023-02-24 14:57:59 +05:00
Andrey Rakhmatullin 2970b35cff Merge remote-tracking branch 'origin/master' into mypy-imports 2023-02-24 02:35:09 +04:00
Andrey Rakhmatullin 3dd9d71c32 Correctly wrap connection errors into ResponseFailed. 2023-02-24 01:13:48 +04:00