RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 149 MiB
Commit Graph

1843 Commits

Author SHA1 Message Date
Andrey Rakhmatullin f422861ef4 Add more tests for spider middlewares. 2021-03-25 20:49:12 +05:00
Andrey Rakhmatullin a6034f942b Add tests for _AsyncCooperatorAdapter. 2021-03-23 22:47:37 +05:00
Andrey Rakhmatullin c51ec1ae1c Drop process_iterable_helper, add _process_iterable_universal. 2021-03-23 22:47:30 +05:00
Eugenio Lacuesta 72e8cea8af
Avoid exceptions in is_generator_with_return_value (#4935) 2021-03-22 11:51:11 -03:00
Adrián Chaves ec5a7918ec
Include Content-Length in HTTP/1.1 responses (#5057) 2021-03-22 11:25:40 +01:00
Mikhail Korobov 0e7ee1256c
Merge pull request #5036 from dmiwell/urllength-dont-skip-silently 
UrlLengthMiddleware: don't skip silently
 2021-03-20 00:55:33 +05:00
Adrián Chaves 308a58aa27
Update CI to support Twisted 21.2.0 (#5027) 2021-03-19 18:39:44 +01:00
Adrián Chaves 8e73e1dfb5 upper-constraints.txt: restrict botocore further 2021-03-18 23:42:29 +01:00
Adrián Chaves 1d83697990
Merge pull request #4769 from scrapy/http2 
Implement experimental HTTP/2 support
 2021-03-18 22:18:12 +01:00
Dmitriy Pomazunovskiy 9cc4513bd6 simpler stats access 2021-03-15 21:38:03 +06:00
Dmitriy Pomazunovskiy 0f254a6afb Test from_settings 2021-03-12 17:11:50 +06:00
Dmitriy Pomazunovskiy d4b2b61255 Use from_settings for backward compatibility 2021-03-12 16:59:37 +06:00
Dmitriy Pomazunovskiy 6e5ea7924c Log skipped urls by length to INFO, add skipped stats 2021-03-12 11:08:41 +06:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Adrián Chaves 5b2d3e17a3
Merge branch 'master' into http2 2021-03-09 14:59:17 +01:00
Eugenio Lacuesta 9e62355271
Allow logger/stats customization in get_retry_request 2021-03-02 12:12:22 -03:00
Adrián Chaves 386e2a51ae tests/test_downloader_handlers_http2.py: fix style issue 2021-02-24 10:41:01 +01:00
Adrián Chaves 12064d799b HTTP/2: improve header handling 2021-02-24 10:37:38 +01:00
Adrián Chaves 5101094207 HTTP/2: test a CONNECT request 2021-02-24 07:33:39 +01:00
Adrián Chaves 7605f19ec4 HTTP/2: test 2 concurrent requests to the same domain 2021-02-23 05:54:48 +01:00
Eugenio Lacuesta 6326178bc5
http2: acceptable protocol update, tests (#4994) 2021-02-22 16:50:51 +01:00
Adrián Chaves 1f7665c4cf Silence a PyLint check on a mistake made for testing purposes 2021-02-22 16:48:10 +01:00
Adrián Chaves 722a33a2ac Fix style issues 2021-02-22 16:42:38 +01:00
Adrián Chaves 80f5003c88 Add tests for get_retry_request 2021-02-22 16:38:38 +01:00
Andrey Rakhmatullin 40eab1d473 Drop a duplicate import. 2021-02-18 19:56:12 +05:00
Andrey Rakhmatullin 58706c6598 Merge branch 'asyncdef-tests' into asyncio-parse-asyncgen-proper-rebased 2021-02-18 17:20:25 +05:00
Andrey Rakhmatullin 49af7c4c8b Drop pytest-twisted, use Scrapy code to install the reactor. 2021-02-18 17:10:30 +05:00
Eugenio Lacuesta e80f37bd3f
Test http2 agent for unsupported scheme 2021-02-17 16:34:29 -03:00
Eugenio Lacuesta ac82a4a8a0
Merge remote-tracking branch 'upstream/master' into http2 2021-02-17 16:33:57 -03:00
Andrey Rakhmatullin abbbfbbb38 Add tests for deferred_f_from_coro_f. 2021-02-12 22:41:16 +05:00
Andrey Rakhmatullin a7557f2a49 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-02-11 16:09:14 +05:00
Andrey Rakhmatullin 54fd371481 Skip uvloop 0.15.0+ on py36. 2021-02-11 14:24:11 +05:00
Adrián Chaves 9ac5b1d021 Adjust test constraints 2021-02-09 09:13:25 +01:00
Adrián Chaves bb72bba178 tox: apply upper constraints to all non-pinned package installations 2021-02-09 09:13:25 +01:00
Adrián Chaves 45345ba6b5 Use constraints.txt to limit pip resolver backtracking 2021-02-09 09:13:25 +01:00
Andrey Rakhmatullin 5cf403295d Remove a duplicate definition. 2021-02-05 19:40:14 +05:00
Andrey Rakhmatullin 2152a2a508 Add main infrastructure for async callbacks. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin d66d52d3ed Add process_iterable_helper. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin 7e9f498e00 Add MutableAsyncChain. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin acff1eb496 Add aiter_errback. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin 76abcedaf4 Add as_async_generator. 2021-02-05 16:21:56 +05:00
Adrián Chaves 248800328c Fix test_pinned_twisted_version 2021-02-03 21:13:43 +01:00
Adrián Chaves c8d8b180bf Merge remote-tracking branch 'upstream/master' into http2 2021-02-03 09:37:35 +01:00
Eugenio Lacuesta 80db569aea
Migrate CI to GitHub actions (#4924) 2021-01-01 19:13:39 -03:00
Eugenio Lacuesta e494a3f733
protocol attribute for h2 responses 2020-12-31 11:50:15 -03:00
Eugenio Lacuesta d698b5147b
Merge branch 'master' into http2 2020-12-31 11:13:25 -03:00
Eugenio Lacuesta b83a1a6fbf Disable test under pypy 2020-12-21 17:21:08 +01:00
Adrián Chaves 0567fdc56e Merge remote-tracking branch 'upstream/master' into tox-pip20.2 2020-12-21 11:02:05 +01:00
Adrián Chaves d0af008608
Merge pull request #4878 from elacuesta/response-protocol-attribute 
Response.protocol attribute
 2020-12-16 15:27:40 +01:00
Adrián Chaves f6879c681e SCRAPY_PINNED → _SCRAPY_PINNED 2020-11-23 23:53:03 +01:00
Adrián Chaves 0dc3e6350c Add a test to check the Twisted version in pinned environments 2020-11-23 22:10:45 +01:00
Eugenio Lacuesta 18b05af877
Remove tests/test_utils_http.py 2020-11-23 16:18:58 -03:00
Eugenio Lacuesta 51ca4d0138
Remove deprecated scrapy.utils.gz.is_gzipped function 2020-11-23 15:47:08 -03:00
Adrián Chaves 07379cf9b7
Merge branch 'master' into allow-customizing-export-column-names 2020-11-20 20:06:18 +01:00
Eugenio Lacuesta 2405df49f1
Add tests for Response.protocol=None 2020-11-16 12:50:33 -03:00
joaquin garmendia 85604e1078
Add failed and success count stats to feedstorage backends (#4850) 2020-11-11 21:16:01 +01:00
Eugenio Lacuesta 5e9a99e6a1
Reponse.protocol as string 2020-11-11 11:15:29 -03:00
Eugenio Lacuesta 5b6b56240c
Test Response.protocol attribute 2020-11-11 01:08:37 -03:00
Eugenio Lacuesta c20b34269f
Remove unnecessary pytest-azurepipelines package (#4876) 2020-11-10 15:35:09 -03:00
Mikhail Korobov 91a8451c98
Merge pull request #4874 from stummjr/fix-missing-fstring-prefix-genspider 
Fix incorrect output on the genspider command
 2020-11-10 17:23:28 +05:00
Eugenio Lacuesta b20cfef1e5
Remove unnecessary line from test 2020-11-09 13:58:52 -03:00
Eugenio Lacuesta 7e98a76ac4
Use deferred_from_coro in asyncio test 2020-11-09 12:17:15 -03:00
Valdir Stumm Junior a2c4a7f920 Add missing f-string prefix to genspider output 2020-11-08 20:14:59 -03:00
Adrián Chaves 4b28da4333 Fix syntax error 2020-11-06 16:46:22 +01:00
Adrián Chaves f3064254ce Merge branch 'patch-2' of github.com:GeorgeA92/scrapy into patch-2 2020-11-06 16:42:58 +01:00
Adrián Chaves ee98771fa7 Remove unused variable 2020-11-06 16:42:32 +01:00
Adrián Chaves 1941f607ca Skip 2 additional tests with older Twisted versions 2020-11-06 16:25:56 +01:00
Eugenio Lacuesta 3095d39740
Test: disable asyncio reactor on Windows for Py>=3.8 2020-11-06 12:16:18 -03:00
Adrián Chaves 5d2a9cf5b1
Merge branch 'master' into patch-2 2020-11-06 14:17:54 +01:00
Adrián Chaves a3e53027ec Test HttpCompressionMiddleware subclasses with custom, parameterless __init__ 2020-11-06 14:16:26 +01:00
Adrián Chaves fea5a11899 Also skip test_asyncdef_asyncio on old Twisted versions 2020-11-06 13:00:42 +01:00
Adrián Chaves ea851b910e Clean up Twisted version check 2020-11-06 13:00:31 +01:00
Eugenio Lacuesta 5b5478ae9d
Call asyncio.get_event_loop when installing the asyncio reactor 2020-11-05 14:01:34 -03:00
Adrián Chaves 906626cf0b Skip MiddlewareUsingCoro::test_asyncdef on asyncio and old Twisted 2020-11-04 21:50:12 +01:00
Adrián Chaves 8e7b756727 Solve Flake8-reported issues 2020-11-04 21:26:55 +01:00
Adrián Chaves 7327145bf3 Remove mitmproxy from pinned environments 2020-10-30 21:34:15 +01:00
Georgiy Zatserklianyi a5872a0fad
Fix output file overwrite with -O (FeedExporter updated) (#4859) 2020-10-30 19:36:39 +01:00
Eugenio Lacuesta 585e4a8aee
Replace local server address 2020-10-15 09:07:34 -03:00
GeorgeA92 872b2e4ce4 testcase added for COMPRESSION_ENABLED setting 2020-10-13 18:41:58 +03:00
GeorgeA92 d32d0d2739 testcase added for HttpCompressionMiddleware with no stats 2020-10-13 18:36:41 +03:00
Eugenio Lacuesta 868826b346
CachingHostnameResolver tests 2020-10-09 10:38:07 -03:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header 
Do not process cookies from headers
 2020-10-08 20:20:32 +05:00
Andrey Rahmatullin 45c06cfd80
Merge pull request #4831 from starrify/downloadermw-support-zstd 
Adding support for zstd in HttpCompressionMiddleware
 2020-10-08 14:08:18 +05:00
Adrián Chaves 269fe35d6d
Merge branch 'master' into http2 2020-10-06 21:05:10 +02:00
P. Chen 156bb0a1d4
Fixing the minor typo on test file path in tests/test_downloadermiddleware_httpcompression.py 2020-10-06 19:53:40 +01:00
Adrián Chaves e40788153c Do not consider about: URLs invalid 2020-10-06 19:13:29 +02:00
GeorgeA92 6050604f62 httocompression/response_bytes tests added 2020-10-06 18:59:57 +03:00
Andrey Rahmatullin b1255b016a
Merge pull request #4746 from Gallaecio/fix_iternodes 
Fix iternodes
 2020-10-06 20:14:04 +05:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
P. Chen 6032a9a310
Minor adjustment to the test case in tests/test_downloadermiddleware_httpcompression.py 2020-10-05 23:55:48 +01:00
P. Chen 50e1f35d1f
Adding test cases for the zstd content encoding 2020-10-05 23:43:12 +01:00
Eugenio Lacuesta 797a6690c0
Tests: use classes instead of paths in settings (#4817) 2020-10-01 23:11:11 -03:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Mikhail Korobov f173af6b9d
Merge pull request #4778 from drs-11/unparseable-no-proxy-fix 
Check for unparseable no_proxy values
 2020-10-01 23:57:00 +05:00
Akshay Sharma 159e2b2e2f
allowing to run .pyw files (#4646) 
* allow .pyw in scrapy/commands/runspider.py

* aesthetics

* added tests for '.pyw'

* created class for testing .pyw files

* name=None parameter in get_log

* small fix

* .pyw tests for non-windows

* used @skipIf for tests

* two more tests skipped
 2020-10-01 23:53:08 +05:00
Eugenio Lacuesta 744f352d09
Do not process cookies from headers 2020-10-01 14:52:23 -03:00
Eugenio Lacuesta 894b509d7a
Crawl rule: remove deprecated code 
Remove the compatibility layer that handles 'process_request'
methods that do not receive a 'response' parameter
 2020-09-29 23:37:28 -03:00
Adrián Chaves 6ef7c44061 Fix timezone test issue 2020-09-22 12:45:21 +02:00
GeorgeA92 c22e810658 httocompression tests added 2020-09-22 07:47:37 +03:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00
Adrián Chaves c3b740f078 Use mocking for tests/test_pipeline_files.py::TestS3FilesStore::test_persist 2020-09-21 23:25:37 +02:00
Adrián Chaves 35726da434 tests/test_feedexport.py: remove unused import 2020-09-21 22:55:25 +02:00
Adrián Chaves 17e135377a Use mocking for tests/test_feedexport.py::BatchDeliveriesTest::test_s3_export 2020-09-21 22:54:39 +02:00
Adrián Chaves 56f05fb164 Use mocking for tests/test_feedexport.py::S3FeedStorageTest::test_store 2020-09-21 22:01:09 +02:00
Adrián Chaves 008cf1c75e Remove a test that has never been executed in Python 3 2020-09-21 20:45:21 +02:00
Andrey Rahmatullin 5e997587d9
Remove dead boto2 code, deprecate is_botocore() (#4776) 2020-09-20 15:06:46 +02:00
Eugenio Lacuesta 6e8d20a07a
HTTP/2: add some type hints (#4785) 2020-09-16 09:57:07 +02:00
drs-11 82ba7c8b52 created separate test for invalid no-proxy values 2020-09-10 20:56:39 +05:30
drs-11 959222df7e check for unparseable no_proxy values 2020-09-05 21:32:05 +05:30
Andrey Rahmatullin c1cc3f2f42
Merge pull request #4761 from Gallaecio/on-the-fly-certificates 
Generate localhost keys for tests on the fly
 2020-09-02 13:37:44 +05:00
Adrián Chaves 9731d91f34
Merge branch 'master' into fix_iternodes 2020-09-01 12:38:37 +02:00
Eugenio Lacuesta 307e35c664
Improve check for invalid cookie in CookiesMiddleware (#4772) 2020-09-01 11:04:00 +02:00
Adrián Chaves 8a3ba34a75 Merge remote-tracking branch 'upstream/master' into http2 2020-08-31 16:43:32 +02:00
Aditya eff33a2e79 fix(h2): Mockserver test uses H2DownloadHandler 2020-08-30 23:55:17 +05:30
maranqz 71d2c2f1a3 improve view of dict 2020-08-30 12:43:44 +03:00
maranqz fc3c66ce95 fix tests: 
* FeedExportConfigTestCase.test_feed_complete_default_values_from_settings_empty
 * FeedExportConfigTestCase.test_feed_complete_default_values_from_settings_non_empty
 2020-08-30 11:44:48 +03:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Mikhail Korobov be655b855d
Merge pull request #4765 from scrapy/simplify-test_crawl 
Simplify tests/test_crawl
 2020-08-29 13:31:20 +05:00
Andrey Rakhmatullin 8123c42737 Simplify running spiders in CrawlSpiderTestCase. 2020-08-29 12:01:58 +05:00
Ammar Najjar ffdf6fe100
use f-strings for the newly merged code from master 
Issue: #4324
 2020-08-29 07:21:48 +02:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Mikhail Korobov 7e6476ff0a
Merge pull request #4764 from scrapy/test_crawl-move-py36 
Merge back tests/py36/_test_crawl.py.
 2020-08-29 00:12:53 +05:00
Andrey Rahmatullin 59a0157ef1
Merge pull request #4722 from Gallaecio/umask 
Do not let umask affect the permissions of startproject-generated files
 2020-08-28 18:41:02 +05:00
Andrey Rakhmatullin de640f41ec Merge back tests/py36/_test_crawl.py. 2020-08-28 18:27:36 +05:00
Ammar Najjar 0e57918231
test(Slot): cover __repr__ 
Issue: #4324
 2020-08-28 13:58:32 +02:00
Mikhail Korobov 64905e3397
Merge pull request #4691 from elacuesta/typing-spider-attributes 
Typing: annotate a few Spider attributes
 2020-08-28 00:03:40 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Adrián Chaves dd378b4bb1 Generate localhost keys for tests on the fly 2020-08-27 11:33:18 +02:00
Mikhail Korobov a6b67cf4a1
Merge pull request #4755 from maranqz/csv-item-exporter-errors 
Add errors parameter for CsvItemExporter with tests
 2020-08-27 12:51:53 +05:00
Ammar Najjar 5ab1a318e8
test: list appears in ValueError Exception message 
Issue #4324
 2020-08-26 15:11:46 +02:00
Eugenio Lacuesta a8114d3731
Typing: annotate a few Spider attributes 2020-08-26 09:00:36 -03:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Mikhail Korobov 0ccaf89a6f
Merge pull request #4756 from ivanprado/master 
Support for delegated methods as callbacks
 2020-08-26 00:10:43 +05:00
Adrián Chaves 2f28cee3ce Add a test to cover searching for a missing node name 2020-08-25 17:49:17 +02:00
Adrián Chaves 6eb1fc4ab5 Merge remote-tracking branch 'upstream/master' into fix_iternodes 2020-08-25 17:01:54 +02:00
Andrey Rahmatullin 067125c303
Merge pull request #4752 from drs-11/master 
DOWNLOAD_MAXSIZE logger level changed from Error to Warning
 2020-08-25 18:35:50 +05:00
Adrián Chaves 39affea93c
Fix style issues 2020-08-25 13:57:48 +02:00
maranqz a2d6fa5adc Add errors parameter for CsvItemExporter with tests 2020-08-25 13:34:43 +03:00
Iván de Prado 3e726b9df7 Support for delegated methods as callbacks 
It can be useful to structure the spiders code around some helper classes.
 2020-08-25 11:31:20 +01:00
Andrey Rahmatullin 64e0ea4ee5
Merge pull request #4735 from wRAR/ciphers-tests 
Re-enable TLS 1.2 in cipher tests.
 2020-08-25 11:48:31 +05:00
WinterComes 2d8ec9d44f Change DOWNLOAD_MAXSIZE logger level from Error to Warning 2020-08-24 20:37:40 +05:30
Aditya 26d344b762 Merge branch 'http2' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-24 14:58:23 +05:30
Eugenio Lacuesta f1250177dc
Remove Python 3.5 from CI (#4743) 2020-08-22 09:33:35 +02:00
Adrián Chaves e90be0d8a5 Mark the new test as xfail for xmliter_lxml 2020-08-21 14:09:52 +02:00
Adrián Chaves d711eca4d9 Merge remote-tracking branch 'upstream/master' into fix_iternodes 2020-08-21 13:52:52 +02:00
Grisha Temchenko d68aab992e
Smarter generator check for combined return/yield statements (#4721) 2020-08-20 10:22:07 -03:00
sakshamb2113 42383cc267
Add a setting to customize the asyncio event loop (#4414) 2020-08-19 09:18:14 +02:00
Aditya a87ab71d10 refactor(http2): metadata for Stream 
- Add Note about HTTP/2 Cleartext not supported in settings.rst
 2020-08-18 05:52:40 +05:30
Andrey Rakhmatullin d9e69bfb51 Re-enable TLS 1.2 in cipher tests. 2020-08-17 19:46:24 +05:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Mikhail Korobov 282a6d4fc1
Merge pull request #4727 from wRAR/mitmproxy-ca 
Use a longer key in mitmproxy-ca.pem.
 2020-08-17 14:06:58 +05:00
Mikhail Korobov 61459de476
Merge pull request #4688 from ashellunts/obsolete-s3feednokeys 
Remove obsolete S3FeedStorage instancing without AWS credentials
 2020-08-17 13:58:32 +05:00
Mikhail Korobov 5ac135f036
Merge pull request #4718 from Gallaecio/support-pinned-3.6 
Upgrade minimum dependencies for Python 3.6 support
 2020-08-17 13:54:45 +05:00
Ajay Mittur a8e08d51cd
Check if file is already present on running `scrapy genspider` and terminate if so (#4623) 2020-08-17 10:45:52 +02:00
Eugenio Lacuesta 2aa4f3cbf9
Conditional request attribute binding for responses (#4632) 2020-08-17 10:39:59 +02:00
adityaa30 75fe3d1365 fix: increase timeout to 0.5 seconds 
- In Windows specifically the reactor was left unclean by the
  HostnameEndpoint due to the tearDown method of
  test_downloader_handlers.py::HttpTestCase due to
  which the following 2 tests were failing:
  1. test_timeout_download_from_spider_server_hangs
  2. test_timeout_download_from_spider_nodata_rcvd
- Increasing the timeout fixed the test (in local)
 2020-08-17 04:18:03 +05:30
Aditya d97cf973dd Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-16 11:31:10 +05:30
Aditya af73f141b2 refactor: move all http2 tests in separate files 2020-08-16 11:26:32 +05:30
Andrey Rakhmatullin 756c368a6b Use a longer key in mitmproxy-ca.pem. 2020-08-14 22:09:24 +05:00
Adrián Chaves 24ba5a71ac Maybe the problem is not in the code after all 2020-08-13 06:35:09 +02:00
Adrián Chaves 5f4df622a1 test_utils_iterators.py: support Windows the right way 2020-08-13 05:41:06 +02:00
Adrián Chaves 4c0afb606c Update permission expectations 2020-08-12 17:45:26 +02:00
Adrián Chaves 125a058340 Do not let umask affect the permissions of startproject-generated files 2020-08-12 17:07:21 +02:00
Adrián Chaves b1de55d37d Fix marker syntax 2020-08-12 12:34:40 +02:00
Adrián Chaves 8e393a0b21 Do not change the mitmproxy version for no-3.6 Python versions 2020-08-12 12:29:51 +02:00
Adrián Chaves 394631fc0a Restore 3.5 support for mitmproxy-based tests 2020-08-12 12:08:09 +02:00
Ajay Mittur 1c4b4cc6b0
Support defining file path based on item in media pipelines (#4686) 2020-08-11 14:12:44 +02:00
Adrián Chaves 5138f9a965 Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-08-11 13:42:05 +02:00
Adrián Chaves aefd43a6c6 Upgrade minimum dependencies for Python 3.6 support 2020-08-11 12:52:54 +02:00
Aditya c67d6dea31 fix: H2 docs, NotImplementedError for H2 Tunnel 2020-08-11 04:39:41 +05:30
Andrey Rahmatullin 0cf1340c29
Merge pull request #4663 from ajaymittur28/scrapy-check-test 
Add Tests to `scrapy check` command
 2020-08-10 16:18:29 +05:00
Aditya e0c3019d90 fix: ScrapyProxyH2Agent 
- add required test cases

BREAKING CHANGES
Presently the tests (in test_downloader_handlers.py)
1. test_download_without_proxy
2. test_download_with_proxy_https_timeout

collide with each other when run together. However, if both of the tests
are ran individually then both pass.
 2020-08-09 16:19:35 +05:30
Jose Galdos 13181ba788 Improve http status all on http error middleware 2020-08-06 20:43:16 -05:00
Marc Hernández 336f19f5cc
Change super syntax (#4707) 2020-08-04 20:42:01 +02:00
Artur Shellunts 5ec66be4a4
Merge branch 'master' into obsolete-s3feednokeys 2020-07-30 22:18:00 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Aditya a3fecaf07f test: fix host-name H2DownloadHandler tests 2020-07-30 15:45:27 +05:30
Mikhail Korobov 5e2d1bd187
Merge pull request #4434 from BroodingKangaroo/ISSUE-4250-add_batch_deliveries 
Feed exports: add batch deliveries
 2020-07-29 23:03:28 +05:00
Aditya e8342996f6 test: H2DownloadHandler 
Following tests are skipped as Content-Length header not matching the
data received is considered as a ProtocolError
- test_download_broken_content_cause_data_loss
- test_download_broken_chunked_content_cause_data_loss
- test_download_broken_content_allow_data_loss
- test_download_broken_chunked_content_allow_data_loss
- test_download_broken_content_allow_data_loss_via_setting
- test_download_broken_chunked_content_allow_data_loss_via_setting

BREAKING CHANGES
The following tests currently fail
- test_content_length_zero_bodyless_post_request_headers
- test_host_header_seted_in_request_headers
- test_download_with_maxsize_very_large_file
 2020-07-29 13:51:01 +05:30
Aditya 92bec38591 feat: MethodNotAllowed405, Content-Length header 
- add tests to check for Content-Length header
- raise MethodNotAllowed405 when remote send 'HTTP/2.0 405 Method Not
Allowed'
 2020-07-29 13:49:14 +05:30
Eugenio Lacuesta 5265853937
Use ItemAdapter.field_names when writing header in CsvItemExporter (#4668) 2020-07-28 11:15:14 +02:00
Aditya 3685e99cca test: http2 connection timeout 2020-07-22 14:47:20 +05:30
Aditya 316620b517 chore: pass spider as argument for request method 
- download_maxsize and download_warnsize can now be extracted from the
spider directly and passed to the stream
- remove `partial` flag from the response as per RFC 7540 - Section
8.1.2.6
 2020-07-22 13:57:49 +05:30
Artur Shellunts 430d22e46e Remove not used import warnings 2020-07-21 23:39:04 +02:00
nyov 2829cd4268 Allow use without credentials 2020-07-21 23:38:02 +02:00
nyov 98e8086d1b Adapt S3FeedStorage testcase 2020-07-21 23:38:02 +02:00
nyov ece4fa6c7c Fix ignored testcase: boto is never installed 2020-07-21 23:38:01 +02:00
Andrey Rahmatullin f3372a3753
Merge pull request #4254 from elacuesta/spider.parse 
Change Scraper API to call internal `_parse` method
 2020-07-21 17:37:54 +05:00
Akshay Sharma de297a3a16
enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 (#4403) 
* changed ie. -> i.e.(spelling error) on lines 667, 763 (issue scrapy#4332)

* updated all text files for issue #4332 (ie. -> i.e.)

* Apply ie. → i.e. in source comments

* ie → e.g.

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* error handling

* error handlingy

* raise ValueError

* added in-built function for version comparison

* recommit changes

* changed check -> parse

* version comparison -> parse_version

* added scrapy/utils/display.py in pytest.ini

* Trigger

* Add simple test for scrapy.utils.display._colorize

* Flake8: E501 for tests/test_utils_display.py

* assertEquals -> assertEqual

* Normal formatter for all platforms

* separate test for windows

* all curses under try block

* added global TestStr

* more test added

* small fix

* covering exceptions

* windows test failing

* Refactor output color handling

* Fix pprint test

* fix flake8

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2020-07-20 17:23:38 +05:00
BroodingKangaroo 3e0492741d Another try to fix test errors on Windows 2020-07-19 00:10:29 +03:00
BroodingKangaroo 86f7ac2f2b Try to fix error at Windows 2020-07-17 18:22:47 +03:00
Artur Shellunts 62a4ede5e9
Remove deprecated classes BaseSgmlLinkExtractor, RegexLinkExtractor and SgmlLinkExtractor (#4356) 2020-07-17 12:40:04 +02:00
Adrián Chaves d29bec60d7
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest) (#4504) 
* Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest)

* Log a detailed error message to discover why MockServer is not working

* Go for all lines!

* Disable tests based on mitmproxy while running on PyPy

* Fix test_get_func_args for PyPy 3.6+

* Make testPayloadDefaultCiphers work regardless of OpenSSL default ciphers

* Crossing fingers…

* Rename: testPayloadDefaultCiphers → testPayloadDisabledCipher

* Test the PyPy version currently documented as the minimum required version

* Fix the PYPY_VERSION tag

* Update the documentation about supported PyPy versions

* Also test the latest 3.5 Python version with PyPy

* Fix the PYPY_VERSION value for the latest 3.5 version

* Use pinned dependencies for asyncio and PyPy tests against oldest supported Python versions

* Fix PyPy installation for the pypy3-pinned Tox environment

* Try installing Cython

* Maybe PyPy requires lxml 3.6.0?

* install.rst: minor clarification

* lxml 4.0.0 is required on PyPy

* Require setuptools 18.5+

* Revert "Require setuptools 18.5+"

This reverts commit 017ec33ac2.

* Maintain lxml as a dependency if setuptools < 18.5 is used
 2020-07-17 02:19:24 +05:00
Mikhail Korobov 9a74a71c63
Merge pull request #4682 from noviluni/remove_python2_reminiscence 
remove python 2 reminiscence in cookies
 2020-07-16 23:58:58 +05:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
Mikhail Korobov 0f2f1acf04
Merge pull request #4458 from scrapy/azure-pipelines 
Set up CI with Azure Pipelines
 2020-07-16 18:09:31 +05:00
Marc 0e0d1ad643 remove python 2 reminiscence in cookies 2020-07-16 14:25:46 +02:00