RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 149 MiB
Commit Graph

1843 Commits

Author SHA1 Message Date
D R Siddhartha 43ea21e830
Feed post-processing plugin support (#5190) 2021-08-24 11:48:01 +02:00
Matsievskiy S.V 3f635eb683
Extract domain from genspider URL (#4439) 2021-08-24 11:05:50 +02:00
Andrey Rahmatullin ada539a63a
Merge pull request #4676 from Gallaecio/startproject-existing-folder 
Fix startproject failing for existing folders
 2021-08-24 12:18:49 +05:00
Andrey Rahmatullin 731f2d3085
Merge pull request #4178 from elacuesta/remove_spider_make_requests_from_url 2021-08-20 16:16:25 +05:00
Andrey Rakhmatullin 5d4b232de8 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-08-16 20:20:43 +05:00
Andrey Rahmatullin 2814e0e197
Disable builtin middlewares in spider middleware tests. (#5229) 2021-08-16 13:22:01 +02:00
Andrey Rakhmatullin b63369c148 Rename tests/requirements-py3.txt to tests/requirements.txt. 2021-08-11 20:02:45 +05:00
Andrey Rahmatullin 0590c3756c
Merge pull request #5221 from wRAR/3.10-tests 
Fixing tests for upcoming 3.10.
 2021-08-11 14:19:51 +05:00
Andrey Rakhmatullin 983b89ad4f Fix SpiderLoaderTest on Python 3.10. 2021-08-11 10:39:23 +05:00
Michel Ace ce9d6c658b
Add more rel_has_nofollow tests 2021-08-10 22:21:51 +02:00
Michel Ace 295f0e2bf5
Make flake8 happy 2021-08-10 21:38:29 +02:00
Michel Ace 07d20a8ce4
Fix test_rel_has_nofollow test 2021-08-10 21:21:43 +02:00
Michel Ace 18b6f30a73
Add test for rel_has_nofollow 2021-08-10 21:13:50 +02:00
Mannan2812 94baa4b272
Fix FileFeedStoragePreFeedOptionsTest fails in CI/CD pipeline (#5198) 2021-08-05 21:23:11 +02:00
Andrey Rakhmatullin 93bf1ae7e3 Fix tests for the 3.10 TypeError message change. 2021-08-03 20:16:59 +05:00
Andrey Rahmatullin b406affd5b
Merge pull request #5210 from laggardkernel/feature/aws-session-token 2021-07-30 15:15:55 +05:00
Eugenio Lacuesta cc89f6be38
Response.attributes (#5218) 2021-07-29 22:12:44 +02:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
Pascal Corpet 70dddfe2b2 Typing: switch to a newer version of MyPy to check types 2021-07-21 17:10:10 +02:00
Andrey Rakhmatullin 7306a81188 Disable builtin middlewares in spider middleware tests. 2021-07-19 20:09:11 +05:00
Adrián Chaves 8a2768561f
Merge branch 'master' into asyncio-parse-asyncgen-proper-rebased 2021-07-16 14:32:15 +02:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Iván de Prado eca641aa3d
Update tests/test_engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:20 +01:00
Iván de Prado Alonso f35970778b Test case for raising CloseSpider on spider idle signal handler 2021-06-29 13:21:38 +01:00
Andrey Rahmatullin 016c7e92d1
Merge pull request #3579 from maramsumanth/duplicate_key 
[MRG+1] Issue #2919: Fix FormRequest.formdata with GET method duplicates same key in query string
 2021-06-15 13:36:53 +05:00
ajaymittur28 7d653288e3 Update unittest 2021-06-14 21:39:18 +05:30
ajaymittur28 857e0e225a Merge branch 'master' into http-proxy-schema 2021-06-14 21:18:49 +05:30
Andrey Rahmatullin 66e2004239
Fix a flake8 problem 2021-06-14 18:58:35 +05:00
Andrey Rahmatullin 16be658a31
Merge branch 'master' into duplicate_key 2021-06-14 18:45:33 +05:00
Andrey Rakhmatullin ecfc924ca8 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-06-04 18:55:32 +05:00
Vostretsov Nikita 23cfdb058e
Reducing amount of warnings during test run (#5162) 
* put flake8 options into separate file to remove pytest warnings

* remove ResourceLeaked warning in pypy

* suppress warnings from twisted

* ignore deprecation warnings here

* ignore deprecation warning in tests of deprecated methods

* ignore deprecation warnings here

* update test classes

* don`t use deprecated method call

* ignore deprecation warnings here

* proper warning class

* more selective ignoring

* Revert "don`t use deprecated method call"

This reverts commit 59216ab560.
 2021-05-28 14:45:06 +05:00
Adrián Chaves c5b1ee8101
Make Twisted[http2] installation optional (#5113) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 09:04:53 +02:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Eugenio Lacuesta 02ae1deaf4
Deprecate unused squeues (#5117) 2021-04-27 09:41:44 -03:00
Eugenio Lacuesta ddea6b7bfa
Scheduler: minimal interface, API docs (#3559) 2021-04-26 16:16:14 -03:00
Eugenio Lacuesta 6837919798
Add peek method to queues (#5112) 2021-04-26 14:55:02 -03:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Andrey Rakhmatullin ffc6f525ce Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-04-14 17:34:58 +05:00
Eugenio Lacuesta d8d1dc5b50
Ignore typing warning in test 2021-04-12 10:43:02 -03:00
Eugenio Lacuesta b6f77806b0
Engine tests: fix item class spider, add minimal type hints 2021-04-09 13:09:47 -03:00
Eugenio Lacuesta 91f8144552
Remove deprecated Spider.make_requests_from_url method 2021-04-07 15:16:55 -03:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Andrey Rakhmatullin 30ed7fa349 Some cleanup, make sync middlewares fail earlier. 2021-04-02 22:20:56 +05:00
Eugenio Lacuesta 9e7cbc05ae
Fix type for urlencode's doseq argument 2021-04-01 15:22:51 -03:00
Mikhail Korobov 12f2006b7b
Merge pull request #4799 from GeorgeA92/patch-2 
httpcompression stats added
 2021-04-01 22:47:24 +05:00
Mikhail Korobov f0c8d31193
Merge pull request #4694 from Jgaldos/improve-httpstatus-all-meta 
Improve http status all on http error middleware
 2021-04-01 22:43:45 +05:00
Mikhail Korobov d0e23486b4
Merge pull request #4902 from Gallaecio/retry-request 
Implement a retry request function
 2021-04-01 22:23:56 +05:00
Andrey Rakhmatullin 6803779eb7 Add more tests for _process_iterable_universal. 2021-03-26 22:29:07 +05:00
Andrey Rakhmatullin 7b515a0485 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-03-25 22:13:49 +05:00
Andrey Rakhmatullin f422861ef4 Add more tests for spider middlewares. 2021-03-25 20:49:12 +05:00
Andrey Rakhmatullin a6034f942b Add tests for _AsyncCooperatorAdapter. 2021-03-23 22:47:37 +05:00
Andrey Rakhmatullin c51ec1ae1c Drop process_iterable_helper, add _process_iterable_universal. 2021-03-23 22:47:30 +05:00
Eugenio Lacuesta 72e8cea8af
Avoid exceptions in is_generator_with_return_value (#4935) 2021-03-22 11:51:11 -03:00
Adrián Chaves ec5a7918ec
Include Content-Length in HTTP/1.1 responses (#5057) 2021-03-22 11:25:40 +01:00
Mikhail Korobov 0e7ee1256c
Merge pull request #5036 from dmiwell/urllength-dont-skip-silently 
UrlLengthMiddleware: don't skip silently
 2021-03-20 00:55:33 +05:00
Adrián Chaves 308a58aa27
Update CI to support Twisted 21.2.0 (#5027) 2021-03-19 18:39:44 +01:00
Adrián Chaves 8e73e1dfb5 upper-constraints.txt: restrict botocore further 2021-03-18 23:42:29 +01:00
Adrián Chaves 1d83697990
Merge pull request #4769 from scrapy/http2 
Implement experimental HTTP/2 support
 2021-03-18 22:18:12 +01:00
Dmitriy Pomazunovskiy 9cc4513bd6 simpler stats access 2021-03-15 21:38:03 +06:00
Dmitriy Pomazunovskiy 0f254a6afb Test from_settings 2021-03-12 17:11:50 +06:00
Dmitriy Pomazunovskiy d4b2b61255 Use from_settings for backward compatibility 2021-03-12 16:59:37 +06:00
Dmitriy Pomazunovskiy 6e5ea7924c Log skipped urls by length to INFO, add skipped stats 2021-03-12 11:08:41 +06:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Adrián Chaves 5b2d3e17a3
Merge branch 'master' into http2 2021-03-09 14:59:17 +01:00
Eugenio Lacuesta 9e62355271
Allow logger/stats customization in get_retry_request 2021-03-02 12:12:22 -03:00
Adrián Chaves 386e2a51ae tests/test_downloader_handlers_http2.py: fix style issue 2021-02-24 10:41:01 +01:00
Adrián Chaves 12064d799b HTTP/2: improve header handling 2021-02-24 10:37:38 +01:00
Adrián Chaves 5101094207 HTTP/2: test a CONNECT request 2021-02-24 07:33:39 +01:00
Adrián Chaves 7605f19ec4 HTTP/2: test 2 concurrent requests to the same domain 2021-02-23 05:54:48 +01:00
Eugenio Lacuesta 6326178bc5
http2: acceptable protocol update, tests (#4994) 2021-02-22 16:50:51 +01:00
Adrián Chaves 1f7665c4cf Silence a PyLint check on a mistake made for testing purposes 2021-02-22 16:48:10 +01:00
Adrián Chaves 722a33a2ac Fix style issues 2021-02-22 16:42:38 +01:00
Adrián Chaves 80f5003c88 Add tests for get_retry_request 2021-02-22 16:38:38 +01:00
Andrey Rakhmatullin 40eab1d473 Drop a duplicate import. 2021-02-18 19:56:12 +05:00
Andrey Rakhmatullin 58706c6598 Merge branch 'asyncdef-tests' into asyncio-parse-asyncgen-proper-rebased 2021-02-18 17:20:25 +05:00
Andrey Rakhmatullin 49af7c4c8b Drop pytest-twisted, use Scrapy code to install the reactor. 2021-02-18 17:10:30 +05:00
Eugenio Lacuesta e80f37bd3f
Test http2 agent for unsupported scheme 2021-02-17 16:34:29 -03:00
Eugenio Lacuesta ac82a4a8a0
Merge remote-tracking branch 'upstream/master' into http2 2021-02-17 16:33:57 -03:00
Andrey Rakhmatullin abbbfbbb38 Add tests for deferred_f_from_coro_f. 2021-02-12 22:41:16 +05:00
Andrey Rakhmatullin a7557f2a49 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-02-11 16:09:14 +05:00
Andrey Rakhmatullin 54fd371481 Skip uvloop 0.15.0+ on py36. 2021-02-11 14:24:11 +05:00
Adrián Chaves 9ac5b1d021 Adjust test constraints 2021-02-09 09:13:25 +01:00
Adrián Chaves bb72bba178 tox: apply upper constraints to all non-pinned package installations 2021-02-09 09:13:25 +01:00
Adrián Chaves 45345ba6b5 Use constraints.txt to limit pip resolver backtracking 2021-02-09 09:13:25 +01:00
Andrey Rakhmatullin 5cf403295d Remove a duplicate definition. 2021-02-05 19:40:14 +05:00
Andrey Rakhmatullin 2152a2a508 Add main infrastructure for async callbacks. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin d66d52d3ed Add process_iterable_helper. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin 7e9f498e00 Add MutableAsyncChain. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin acff1eb496 Add aiter_errback. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin 76abcedaf4 Add as_async_generator. 2021-02-05 16:21:56 +05:00
Adrián Chaves 248800328c Fix test_pinned_twisted_version 2021-02-03 21:13:43 +01:00
Adrián Chaves c8d8b180bf Merge remote-tracking branch 'upstream/master' into http2 2021-02-03 09:37:35 +01:00
Eugenio Lacuesta 80db569aea
Migrate CI to GitHub actions (#4924) 2021-01-01 19:13:39 -03:00
Eugenio Lacuesta e494a3f733
protocol attribute for h2 responses 2020-12-31 11:50:15 -03:00
Eugenio Lacuesta d698b5147b
Merge branch 'master' into http2 2020-12-31 11:13:25 -03:00
Eugenio Lacuesta b83a1a6fbf Disable test under pypy 2020-12-21 17:21:08 +01:00
Adrián Chaves 0567fdc56e Merge remote-tracking branch 'upstream/master' into tox-pip20.2 2020-12-21 11:02:05 +01:00
Adrián Chaves d0af008608
Merge pull request #4878 from elacuesta/response-protocol-attribute 
Response.protocol attribute
 2020-12-16 15:27:40 +01:00
Adrián Chaves f6879c681e SCRAPY_PINNED → _SCRAPY_PINNED 2020-11-23 23:53:03 +01:00
Adrián Chaves 0dc3e6350c Add a test to check the Twisted version in pinned environments 2020-11-23 22:10:45 +01:00
Eugenio Lacuesta 18b05af877
Remove tests/test_utils_http.py 2020-11-23 16:18:58 -03:00
Eugenio Lacuesta 51ca4d0138
Remove deprecated scrapy.utils.gz.is_gzipped function 2020-11-23 15:47:08 -03:00
Adrián Chaves 07379cf9b7
Merge branch 'master' into allow-customizing-export-column-names 2020-11-20 20:06:18 +01:00
Eugenio Lacuesta 2405df49f1
Add tests for Response.protocol=None 2020-11-16 12:50:33 -03:00
joaquin garmendia 85604e1078
Add failed and success count stats to feedstorage backends (#4850) 2020-11-11 21:16:01 +01:00
Eugenio Lacuesta 5e9a99e6a1
Reponse.protocol as string 2020-11-11 11:15:29 -03:00
Eugenio Lacuesta 5b6b56240c
Test Response.protocol attribute 2020-11-11 01:08:37 -03:00
Eugenio Lacuesta c20b34269f
Remove unnecessary pytest-azurepipelines package (#4876) 2020-11-10 15:35:09 -03:00
Mikhail Korobov 91a8451c98
Merge pull request #4874 from stummjr/fix-missing-fstring-prefix-genspider 
Fix incorrect output on the genspider command
 2020-11-10 17:23:28 +05:00
Eugenio Lacuesta b20cfef1e5
Remove unnecessary line from test 2020-11-09 13:58:52 -03:00
Eugenio Lacuesta 7e98a76ac4
Use deferred_from_coro in asyncio test 2020-11-09 12:17:15 -03:00
Valdir Stumm Junior a2c4a7f920 Add missing f-string prefix to genspider output 2020-11-08 20:14:59 -03:00
Adrián Chaves 4b28da4333 Fix syntax error 2020-11-06 16:46:22 +01:00
Adrián Chaves f3064254ce Merge branch 'patch-2' of github.com:GeorgeA92/scrapy into patch-2 2020-11-06 16:42:58 +01:00
Adrián Chaves ee98771fa7 Remove unused variable 2020-11-06 16:42:32 +01:00
Adrián Chaves 1941f607ca Skip 2 additional tests with older Twisted versions 2020-11-06 16:25:56 +01:00
Eugenio Lacuesta 3095d39740
Test: disable asyncio reactor on Windows for Py>=3.8 2020-11-06 12:16:18 -03:00
Adrián Chaves 5d2a9cf5b1
Merge branch 'master' into patch-2 2020-11-06 14:17:54 +01:00
Adrián Chaves a3e53027ec Test HttpCompressionMiddleware subclasses with custom, parameterless __init__ 2020-11-06 14:16:26 +01:00
Adrián Chaves fea5a11899 Also skip test_asyncdef_asyncio on old Twisted versions 2020-11-06 13:00:42 +01:00
Adrián Chaves ea851b910e Clean up Twisted version check 2020-11-06 13:00:31 +01:00
Eugenio Lacuesta 5b5478ae9d
Call asyncio.get_event_loop when installing the asyncio reactor 2020-11-05 14:01:34 -03:00
Adrián Chaves 906626cf0b Skip MiddlewareUsingCoro::test_asyncdef on asyncio and old Twisted 2020-11-04 21:50:12 +01:00
Adrián Chaves 8e7b756727 Solve Flake8-reported issues 2020-11-04 21:26:55 +01:00
Adrián Chaves 7327145bf3 Remove mitmproxy from pinned environments 2020-10-30 21:34:15 +01:00
Georgiy Zatserklianyi a5872a0fad
Fix output file overwrite with -O (FeedExporter updated) (#4859) 2020-10-30 19:36:39 +01:00
Eugenio Lacuesta 585e4a8aee
Replace local server address 2020-10-15 09:07:34 -03:00
GeorgeA92 872b2e4ce4 testcase added for COMPRESSION_ENABLED setting 2020-10-13 18:41:58 +03:00
GeorgeA92 d32d0d2739 testcase added for HttpCompressionMiddleware with no stats 2020-10-13 18:36:41 +03:00
Eugenio Lacuesta 868826b346
CachingHostnameResolver tests 2020-10-09 10:38:07 -03:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header 
Do not process cookies from headers
 2020-10-08 20:20:32 +05:00
Andrey Rahmatullin 45c06cfd80
Merge pull request #4831 from starrify/downloadermw-support-zstd 
Adding support for zstd in HttpCompressionMiddleware
 2020-10-08 14:08:18 +05:00
Adrián Chaves 269fe35d6d
Merge branch 'master' into http2 2020-10-06 21:05:10 +02:00
P. Chen 156bb0a1d4
Fixing the minor typo on test file path in tests/test_downloadermiddleware_httpcompression.py 2020-10-06 19:53:40 +01:00
Adrián Chaves e40788153c Do not consider about: URLs invalid 2020-10-06 19:13:29 +02:00
GeorgeA92 6050604f62 httocompression/response_bytes tests added 2020-10-06 18:59:57 +03:00
Andrey Rahmatullin b1255b016a
Merge pull request #4746 from Gallaecio/fix_iternodes 
Fix iternodes
 2020-10-06 20:14:04 +05:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
P. Chen 6032a9a310
Minor adjustment to the test case in tests/test_downloadermiddleware_httpcompression.py 2020-10-05 23:55:48 +01:00
P. Chen 50e1f35d1f
Adding test cases for the zstd content encoding 2020-10-05 23:43:12 +01:00
Eugenio Lacuesta 797a6690c0
Tests: use classes instead of paths in settings (#4817) 2020-10-01 23:11:11 -03:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Mikhail Korobov f173af6b9d
Merge pull request #4778 from drs-11/unparseable-no-proxy-fix 
Check for unparseable no_proxy values
 2020-10-01 23:57:00 +05:00
Akshay Sharma 159e2b2e2f
allowing to run .pyw files (#4646) 
* allow .pyw in scrapy/commands/runspider.py

* aesthetics

* added tests for '.pyw'

* created class for testing .pyw files

* name=None parameter in get_log

* small fix

* .pyw tests for non-windows

* used @skipIf for tests

* two more tests skipped
 2020-10-01 23:53:08 +05:00
Eugenio Lacuesta 744f352d09
Do not process cookies from headers 2020-10-01 14:52:23 -03:00
Eugenio Lacuesta 894b509d7a
Crawl rule: remove deprecated code 
Remove the compatibility layer that handles 'process_request'
methods that do not receive a 'response' parameter
 2020-09-29 23:37:28 -03:00
Adrián Chaves 6ef7c44061 Fix timezone test issue 2020-09-22 12:45:21 +02:00
GeorgeA92 c22e810658 httocompression tests added 2020-09-22 07:47:37 +03:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00