RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

2349 Commits

Author SHA1 Message Date
Josh Soref d27c611cc0 spelling: referrer 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 6aa5374bd3 spelling: received 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 826e0ee611 spelling: preexisting 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 66ab82f126 spelling: precedence 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 8ae77fdb34 spelling: pipeline 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 1a5cf00db7 spelling: overridden 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref a839b61147 spelling: nonexistent 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref e894db2f3f spelling: laziness 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref a26b6b0607 spelling: indistinct 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Josh Soref 1300c1c881 spelling: children 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Alex 93ace91ace
Merge branch 'scrapy:master' into fix_shell_fetch_alt 2022-12-13 21:50:28 -08:00
Alex c2de9372a2 added unittest for scrapy shell command and fixed flake8 error 2022-12-13 21:37:17 -08:00
Matt Mayfield 40f4b262d2 Fix style errors 2022-12-12 12:36:29 -05:00
Matt Mayfield 8d67a08155 Change test name and add additional tests 2022-12-12 11:55:42 -05:00
Matt Mayfield 66f127eb37 Make test cleaner and more reusable 2022-12-12 11:46:05 -05:00
Laerte Pereira e099572cec
change xfail condition 2022-12-12 09:56:50 -03:00
Laerte Pereira 0dbd1d9b81
use xfail, switch to packaging to parse version 2022-12-12 08:58:40 -03:00
Matt Mayfield 087334009c Call `finish_exporting` even when itemcount == 0 2022-12-11 23:12:41 -05:00
Laerte Pereira 6757973b61
fix test_follow_whitespace_* tests 2022-12-11 09:11:55 -03:00
Emmanuel Rondan 1a3db81492 ignoring E203 flake8 error 
ignoring implicit-str-concat pylint error
fixing pylint comment on test_request_cb_kwargs.py
 2022-11-29 15:06:54 -03:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Andrey Rakhmatullin f85c3f3d68 Add a comment about `tmpname + '^'`. 2022-11-25 18:46:14 +06:00
Laerte Pereira e769532644
Remove `noconnect` deprecate code 2022-11-23 08:30:11 -03:00
Laerte Pereira 1506479672
Remove deprecated test 2022-11-22 10:07:32 -03:00
Laerte Pereira 8e0025f53d
Remove support for override settings with `SCRAPY_` environment variables 2022-11-22 09:38:54 -03:00
Laerte Pereira 1a6408c3fa
Remove `FilteringLinkExtractor` 2022-11-21 07:30:20 -03:00
Laerte Pereira d5b6c236a9
Remove deprecated code (#5719) 2022-11-21 09:32:26 +01:00
GeorgeA92 042012f6bd per slot settings: error calculation metho updated 2022-11-18 14:56:29 +02:00
Andrey Rahmatullin bbe24d79a5
Fix test issues 2022-11-07 17:08:54 +05:00
Andrey Rahmatullin 6c0890ff54
Simplify the changes after the merge 2022-11-07 16:36:54 +05:00
Andrey Rahmatullin b03d84e9fb
Merge branch 'master' into 3689-update 2022-11-07 16:33:21 +05:00
Andrey Rahmatullin 77cd511a5e
Merge pull request #5489 from PluT00/deprecate-NoimagesDrop 
Deprecate scrapy.pipelines.images.NoimagesDrop
 2022-11-05 14:18:19 +05:00
Andrey Rahmatullin 8004075823
Merge pull request #5699 from elacuesta/crawlspider-callback-keyword-arguments 
CrawlSpider: pass cb_kwargs from process_request
 2022-11-02 17:35:03 +05:00
Andrey Rahmatullin a95a338eea
Merge pull request #5599 from tonal/patch-1 
Partial in is_generator_with_return_value
 2022-10-31 20:08:16 +05:00
Eugenio Lacuesta b18560315b
Add tests 2022-10-30 18:28:16 -03:00
Andrey Rakhmatullin b71d0292d5 Add a test for processing partial callbacks. 2022-10-27 18:13:47 +06:00
Godson 3a34fa8399
Get the event loop from event_loop_policy to avoid a deprecation warning (#5689) 2022-10-27 14:02:12 +02:00
Andrey Rakhmatullin ca50af6453 Remove an unused import. 2022-10-27 17:12:26 +06:00
Andrey Rahmatullin 5780ccba55
Merge branch 'master' into deprecate-NoimagesDrop 2022-10-27 15:46:32 +05:00
Adrián Chaves 46dd152b3e Merge remote-tracking branch 'upstream/master' into more-lenient-proxying 2022-10-26 16:55:04 +02:00
Godson-Gnanaraj 830e1c5dd8 Add test for parsing decorated methods 2022-10-26 01:26:54 +05:30
Andrey Rahmatullin b33244e2f0
Fix the flake8 per-file ignore syntax (#5688) 2022-10-21 16:17:04 +02:00
Matthew Donoughe 12a26755ae
improve render_templatefile 2022-10-20 20:14:43 -04:00
Matthew Donoughe c85de90819
fix path to tox.ini 2022-10-20 19:44:35 -04:00
Matthew Donoughe 065db7b566
fix some mypy issues 2022-10-19 08:28:26 -04:00
Matthew Donoughe 93d82648e5
resolve this path too 2022-10-19 08:08:21 -04:00
Matthew Donoughe fb26e6b650
resolve path to search in error 2022-10-18 20:21:11 -04:00
Matthew Donoughe 7daf735f45
downgrade type hint syntax 2022-10-18 20:07:29 -04:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Adrián Chaves 06c8f673af
2.7 release notes (#5680) 
* Fix the display name of documented fingerprinter class methods

* Initial draft for the Scrapy 2.7 release notes

* Update VERSION and PREVIOUS_VERSION references

* Clarify the restrictions lifted for item field output names

* Fix the description of the BOM bug fix

* Fix the note about changes in MIME sniffing

* Fix typo

* Extend highlights

* Fyx typo
 2022-10-17 18:04:29 +05:00
Andrey Rahmatullin ea6315b404
Merge pull request #5679 from wRAR/template-asyncio-reactor 
Change TWISTED_REACTOR in the default template.
 2022-10-15 14:54:59 +05:00
Adrián Chaves 960a7f68f6
Verify that the installed asyncio event loop matches ASYNCIO_EVENT_LOOP (#5529) 
Co-authored-by: Laerte Pereira <5853172+Laerte@users.noreply.github.com>
 2022-10-15 11:27:00 +02:00
Adrián Chaves 75bb516edb Adapt tests to the new value of TWISTED_REACTOR for new projects 2022-10-15 10:26:38 +02:00
Mohammadtaher Abbasi 043575123c
Add async callback support to the parse command (#5577) 2022-10-15 10:11:05 +02:00
Andrey Rakhmatullin 5bf4260679 Update flake8. 2022-10-10 11:14:20 +06:00
Andrey Rakhmatullin eeb199adda Fix flake8 issues in previously ignored files. 2022-10-10 11:10:59 +06:00
Andrey Rahmatullin d96c465dde
Merge pull request #5639 from Laerte/fix/test_batch_path_differ 
fix: `batch_path_differ` test
 2022-10-05 14:44:30 +05:00
Eugenio Lacuesta da8f915091
Adapt for asyng generator callbacks 2022-10-02 17:37:10 -03:00
Eugenio Lacuesta 80194f1c03
CrawlSpider: add support for async def callbacks 2022-10-02 15:22:06 -03:00
Laerte Pereira 3ca7877781
chore: Skip `batch_path_differ` test on Windows 2022-09-29 11:51:50 -03:00
Gábor Lipták 385acd5598
Match pyOpenSSL and service_identity to Twisted 
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
 2022-09-24 21:33:02 -04:00
Adrián Chaves 77c055ee28 Relax Proxy-Authorization restrictions 2022-09-14 14:47:14 +02:00
Mikhail Korobov 90b8503789
Merge pull request #5611 from mtabbasi/issue-5601 
BOM should take precedence over Content-Type header when detecting the encoding

Fixes GH-5601.
 2022-09-13 15:07:50 +05:00
Laerte Pereira 582a6bf6db
refactor: Use `safe_url_string` to standardize url output 2022-09-06 10:03:18 -03:00
Laerte Pereira 1289422284
chore: Skip `test_ipv6_default_name_resolver` test if w3lib version >= 2.0.0 2022-09-06 08:17:58 -03:00
Laerte Pereira f4bcc3e67d
fix: failed tests 2022-09-06 01:15:41 -03:00
Mohammadtaher Abbasi a988c4b78b add test 2022-08-29 17:08:30 +04:30
Adrián Chaves d599fff2b9 Test the life cycle of a storage file 2022-07-29 19:12:20 +02:00
Mikhail Korobov 56e2eeac10 fix typing issues by upgrading mypy 2022-07-27 09:41:12 +05:00
Mikhail Korobov 4dde2f2c36 Merge branch 'master' into asyncio-parse-fixes 2022-07-27 09:32:54 +05:00
Adrián Chaves 00ccb02f1f Merge remote-tracking branch 'upstream/2.6' 2022-07-25 14:13:22 +02:00
Adrián Chaves af7dd16d8d
Merge pull request from GHSA-9x8m-2xpf-crp3 
* Enforce matching proxy request meta and Proxy-Authorization header

* Cover proxy credential security fix in the release notes

* Remove extra empty line

* Reword the security issue description

* Address scenario where Proxy-Authorization is unexpectedly removed by a prior middleware

* Set the release date of Scrapy 2.6.2 and 1.8.3
 2022-07-25 13:15:17 +02:00
Andrey Rakhmatullin b21c16099e Fix flake8 issues. 2022-07-22 19:18:33 +05:00
Andrey Rahmatullin a0681fb811
Merge branch 'master' into asyncio-parse-asyncgen-proper-rebased 2022-07-22 18:30:38 +05:00
Mikhail Korobov d78f505f3d
Merge pull request #5450 from kinoute/master 
fixed detection of extensions like ".tar.gz" in URL
 2022-07-20 01:06:11 +05:00
Mikhail Korobov b103664bf4
Address 2/3 of warnings from tests (#5561) 2022-07-19 17:39:26 +02:00
Andrey Rahmatullin 44580851ff
Merge pull request #5560 from scrapy/non-existing-resolvable 
make Scrapy testing suite more robust in environments where non-existing hosts are resolvable
 2022-07-19 11:28:38 +05:00
Mikhail Korobov 1054689593 remove unused imports 
thanks flake8!
 2022-07-18 23:53:30 +05:00
Mikhail Korobov e248360e6e remove compatibility code from tests for the case dataclasses module is not available 
It was Python 3.6 compat code, and Python 3.6 support is dropped.
 2022-07-18 23:49:08 +05:00
Mikhail Korobov 26c70318cb make Scrapy testing suite more robust in environments where non-existing hosts are resolvable 2022-07-17 16:47:20 +05:00
Mikhail Korobov b51b52ff37
Merge pull request #5526 from mtabbasi/headers-multiple-values 
Merge values of multiple headers with same name
 2022-06-27 23:03:01 +05:00
Mikhail Korobov 500dae82e2
Merge pull request #3696 from Gallaecio/allow-customizing-export-column-names 
Allow customizing export column names
 2022-06-27 00:26:05 +05:00
Adrián Chaves 4ef71829b2
If TWISTED_REACTOR is None, reuse any pre-installed reactor (#5528) 2022-06-17 10:37:27 +02:00
Adrián Chaves ec5cf3e9ce test_feedexport: solve ordered comparison issues 2022-06-17 09:10:18 +02:00
Adrián Chaves bc285f393c Revert "Revert "test_feedexport: remove ordered=False"" 
This reverts commit 516e2d6ec0.
 2022-06-17 09:00:39 +02:00
Adrián Chaves 516e2d6ec0 Revert "test_feedexport: remove ordered=False" 
This reverts commit 24f382fa45.
 2022-06-17 08:55:45 +02:00
Adrián Chaves 3729c6d266 Remove unused import and redundant import 2022-06-17 08:33:34 +02:00
Adrián Chaves 24f382fa45 test_feedexport: remove ordered=False 2022-06-17 08:31:45 +02:00
Adrián Chaves 1b9ed22bec Remove Python < 3.7 leftover 2022-06-17 08:27:17 +02:00
Adrián Chaves 3e994bda45 Update for Python 3.7+ 2022-06-16 20:53:14 +02:00
Kromitvs 9e265a2c1f
Mind body to choose response class in cache, FTP and HTTP/1.0 (#4873) 2022-06-16 20:52:19 +02:00
Adrián Chaves e8503217fb Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2022-06-16 20:19:52 +02:00
Mohammadtaher Abbasi 892c2a4655 delete unnecessary test 2022-06-13 23:46:42 +04:30
Mikhail Korobov de0e2ccd7b
Merge pull request #5451 from FJMonteroInformatica/master 
make test files follow HTML conventions
 2022-06-12 00:46:45 +05:00
Mohammadtaher Abbasi ddfd192b70 add tests for multiple headers with same name 2022-06-11 23:51:34 +04:30
Laerte Pereira 6770d1ec62
chore(tests): Remove validations for unsupported modules versions 2022-06-09 09:08:09 -03:00
Laerte Pereira 407562b38b
Drop Python 3.6 support (#5514) 
* chore: Drop Python 3.6 support

* Attend PR comments

* Tweak versions

* Update dependencies version

* fix: Ubuntu workflow

* fix windows workflow

* chore: Remove comment

* update `install_requires` dependencies versions

* move lxml to main pinned requirements

* Attend code-review comments

* remove non-pinned 3.7 from windows workflow

* simplify condition

* lint

* remove paragraph

* refactor

* remove leftover
 2022-06-09 00:25:03 -03:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Mikhail Korobov 517cbc8d71
Merge pull request #5508 from mtabbasi/thumb_path-access-item 
pass on item to thumb_path function as additional argument
 2022-06-05 03:42:45 +05:00
Mikhail Korobov 99cddecd28
Merge pull request #5497 from AngelikiBoura/issue_3264/command_parser_unhandled_error 
Issue #3264, fix error handling when spider is not matched
 2022-05-28 13:26:23 +05:00
Eugenio Lacuesta 2c65066ad9
Avoid exceptions on copy 2022-05-27 19:56:42 -03:00
Eugenio Lacuesta 1b1d518e2b
Merge remote-tracking branch 'upstream/master' into case-insensitive-dict 2022-05-27 18:45:10 -03:00
Mohammadtaher Abbasi 5c586d78f0 add tests 2022-05-25 23:58:09 +04:30
Andreas Tziortziortziopoulos b5c15d87ff [issue3264] Separate test for not matched spider to a url 2022-05-22 12:19:20 +03:00
Maxime Nannan 078622cfb0
Fix file expiration issue with GCS (#5318) 2022-05-20 08:30:06 +02:00
Eugenio Lacuesta 1c031b8a8d
Underscore CaseInsensitiveDict normkey/normvalue 2022-05-12 13:10:08 -03:00
Eugenio Lacuesta 7297ae566f
Merge remote-tracking branch 'upstream/master' into case-insensitive-dict 2022-05-12 12:43:41 -03:00
GeorgeA92 0ffc52a491 per slot settings: test added (delays for each download slots) 2022-05-11 10:40:12 +03:00
Andreas Tziortziortziopoulos 83c1939281 Issue #3264, fix error handling when spider is not matched 
Changes
Implementation:
- Check whether Spider exists or is None, and if it's None skip execution of start_requests() with non existing Spider
Testing:
- Add a test case with invalid url inside test_command_parse
  Test proves that non-matched Spider does not throw an AttributeError
 2022-05-06 03:59:30 +03:00
AngelikiBoura b2afcbfe2b
Fix typos in three files for Flake8 check (#5487) 
* Fix typos in extensions files

Made some fixes in files memusage.py and statsmailer.py
in order to pass the flake8 check.

* Fix typos in twisted_reactor_custom_settings_same.py

A small change was needed in order for flake8 check
to pass.
 2022-05-05 15:49:52 +02:00
PluT00 cc16af35af Add deprecation warning test for scrapy.pipelines.images.NoimagesDrop 2022-05-03 11:29:21 +03:00
Alex 636127ec1e tests that all CLI help messages don't throw errors 2022-04-22 16:32:58 -07:00
Andrey Rahmatullin 35b44f3b98
Merge pull request #5436 from Gallaecio/fix-crawlerprocess-regression 
CrawlerProcess: initialize the reactor only once
 2022-03-25 13:21:21 +05:00
Andrey Rahmatullin 1e1bf353bf
Merge pull request #5440 from Gallaecio/fix-closespider-exception 
engine: prevent slot method call after unsetting the slot
 2022-03-24 19:43:31 +05:00
FJMonteroInformatica 6a3f2ee687 HTML Conventions 2022-03-17 20:09:56 +01:00
Eugene 9a28eb0bad
Suggest installing the brotli package instead of brotlipy (#4267) 2022-03-17 05:39:54 +01:00
Yann Defretin 5b4b8b6fb1
added test for new url_has_any_extension function 2022-03-16 22:32:05 +01:00
Adrián Chaves c961438d5d tests: cover scenarios of bad results from process_spider_output 2022-03-16 18:45:56 +01:00
Adrián Chaves cb3afcf0e7 Merge remote-tracking branch 'upstream/master' into wrar-asyncio-parse-asyncgen-proper-rebased 2022-03-16 18:08:47 +01:00
Adrián Chaves b59a69be17 Test that a low CLOSEPIDER_TIMEOUT does not raise an exception 2022-03-15 12:10:41 +01:00
Laerte Pereira c1d4be8cb5
Restore ‘-o -’ support (#5445) 2022-03-15 11:30:30 +01:00
Adrián Chaves 96fc4dae15 CrawlerProcess: test a multi-spider scenario 2022-03-02 17:30:05 +01:00
Adrián Chaves 50c8becbe0
Freeze and upgrade CI packages (#5429) 2022-03-01 17:29:08 +01:00
Adrián Chaves e865c4430e
Merge pull request from GHSA-mfjm-vh54-3f96 
* Ignore cookies with a public suffix as domain unless it matches the request domain

* Fix the merge of 1.8.2 release notes

* Re-apply removal of tldextract restriction
 2022-03-01 12:38:19 +01:00
Adrián Chaves 8ce01b3b76
Merge pull request from GHSA-cjvr-mfj7-j4j8 
* Do not carry over cookies to a different domain on redirect

* Cover the cookie-domain redirect fix in the release notes

* Cover 1.8.2 in the release notes

* Fix redirect Cookie handling when the cookie middleware is disabled

* Update the 1.8.2 release date
 2022-03-01 12:26:05 +01:00
Laerte Pereira befb6df119 Remove Python 2 code from WrappedRequest 2022-02-11 06:19:27 -03:00
Andrey Rahmatullin 5d7c0a5f86
Use toscrape.com instead of example.com in test_command_check. (#5407) 2022-02-10 10:50:12 +01:00
Alex e2e2ffd0d1
Move from optparse to argparse (#5374) 2022-02-09 20:52:07 +01:00
Andrey Rakhmatullin 1e1cfc26db Copy resource classes from twisted.web.test.test_webclient. 2022-02-08 21:01:16 +05:00
Laerte 4bda0976b2
Fix csviter call, add parse_rows test (#5394) 2022-02-08 14:57:19 +01:00
Eugenio Lacuesta fca49cca92
Remove deprecated DictItem class 2022-02-06 18:31:55 -03:00
Eugenio Lacuesta c8c1edd43b
Flake8 adjustments 2022-02-06 18:27:41 -03:00
Eugenio Lacuesta bbfa185664
Remove deprecated BaseItem class 2022-02-06 18:12:28 -03:00
Eugenio Lacuesta 55ae2109c9
Remove deprecated TextResponse.body_as_unicode 2022-02-05 13:11:13 -03:00
Laerte fe43411bc4
Fix TypeError on using pathlib.Path as key on FEEDS settings (#5384) 2022-02-04 09:57:57 +01:00
Sixuan (Cherie) Wu 30d5779ea9
Fix FEED_URI_PARAMS: custom params throws KeyError (#4966) 
* fix FEED_URI_PARAMS: custom params throws KeyError closes #4962

* another try FEED_URI_PARAMS

* add warning message and change default function

* Add tests for FEED_URI_PARAMS

* FEED_URI_PARAMS: warn if the params dict has been modified in-place

* [Doc] FEED_URI_PARAMS: modifying params in-place is deprecated

* Remove whileline

* Rename parameters for lambda function

* Type hints for FeedExporter._get_uri_params

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2022-01-28 14:30:30 -03:00
Georgiy Zatserklianyi 4bdaa54af4
response_httprepr memory issue fixed (#4972) 
* response_httprepr replaced by response.body

* unused import deleted

* get_header_size function added

* response size calculation updated

* flake8 codestyle fix

* added counting status code, line breaks to response size

* get_status size: list to tuple, comments added

* test added: comparing new response size counting method with old `len(response_httprepr)`

* downloader stats : unreachable code deleted

* `get_status_size` optimized

* comment added

* tests.test_downloadermiddleware_stats: statement formatting updated

* scrapy.utils.response: `response_httprepr` -> deprecated

* tests.test_downloadermiddleware_stats: flake8 fix
 2022-01-28 18:39:32 +05:00
Andrey Rakhmatullin f789547551 Implement spider middleware iterable upgrade/downgrade. 2022-01-11 18:28:32 +05:00
Andrey Rakhmatullin 8864407c4d Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2022-01-10 19:37:46 +05:00
Andrey Rakhmatullin 3577c00569 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2022-01-03 13:59:26 +05:00
Andrey Rahmatullin c5ab58056c
Set WindowsSelectorEventLoopPolicy on Windows (#5315) 2021-12-31 20:38:10 +01:00
Andrey Rakhmatullin d4565318c7 Fix a reactor test on Windows. 2021-12-23 17:40:31 +05:00
Andrey Rakhmatullin df04e934e6 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-23 17:25:56 +05:00
Andrey Rakhmatullin 9c4bfb4836 Remove an unused import. 2021-12-23 17:17:36 +05:00
Andrey Rakhmatullin 0025134382 Completely skip WindowsRunSpiderCommandTest outside Windows. 2021-12-23 16:45:17 +05:00
Andrey Rakhmatullin ebcafdf4a9 Add tests for TWISTED_REACTOR in custom_settings. 2021-12-23 16:35:26 +05:00
Andrey Rakhmatullin 041699b54c Remove tests that want to modify the test process reactor. 2021-12-23 16:14:47 +05:00
Andrey Rakhmatullin 60c8838554 Move installing the reactor from CrawlerProcess to Crawler. 2021-12-23 16:07:18 +05:00
Andrey Rahmatullin 9ec60e8e4a
Merge pull request #5320 from zessx/5319-oib-base-tag 2021-12-21 17:49:31 +05:00
Alex c316ca45a5
Use augmented assignment statements (#5322) 2021-11-16 10:20:56 +01:00
Samuel Marchal 75ed765476 Test coverage for open_in_browser base tag injection (#5319) 2021-11-15 16:44:56 +01:00
azzamsa 55cce25a79 test: `test_format_engine_status` 2021-11-05 03:02:02 +07:00
Andrey Rakhmatullin c105a7a85f Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-10-22 21:52:47 +05:00
Andrey Rahmatullin 144d1eb834
Add Deferred-to-Future helpers (#5288) 2021-10-22 18:46:01 +02:00
Adrián Chaves d774d6a9c4
Remove unused variable 2021-10-15 17:25:22 +02:00
Erik Kemperman ca320feb2a
Add LOG_FILE_APPEND to settings 2021-10-15 16:36:41 +02:00
Andrey Rahmatullin 5b13bfd14a
Merge pull request #5269 from Ankur19/allow-non-text-response-types-in-ItemLoader 
Fix bug #5145 - Removing Selector for Response's that are not Http or Xml
 2021-10-14 16:19:54 +05:00
Andrey Rakhmatullin a642b73e1a Remove an unused import. 2021-10-13 19:22:22 +05:00
Andrey Rakhmatullin e5b057cfd4 Don't use a HTTP request in a case that will not be awaited. 2021-10-13 19:07:53 +05:00
Andrey Rakhmatullin 66d8a71920 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-10-11 20:24:22 +05:00
Kian-Meng, Ang 3a263280ba Fix typos 2021-10-11 22:32:42 +08:00
ankur19 6fbd6f941f Fix issue#5145 
Fix condition for failing tests

set Selector to None on AttributeError

Add test and remove unused imports

Fix imports
 2021-10-10 15:55:22 -04:00
Georgiy Zatserklianyi d3f1bf79e8
Use f-strings where appropriate (#5246) 2021-10-07 16:27:20 +02:00
Eugenio Lacuesta 029cab72e8
[CI] fix pypy test (#5264) 2021-10-06 19:34:09 +02:00
Laerte 6c858cec91
Cookies: Cast primitive types to str (#5253) 
* cast primitive types to str

* add tests
 2021-10-06 12:32:04 -03:00
Andrey Rakhmatullin b081f18a2f Add http_auth_domain to HttpAuthMiddleware. 2021-10-06 12:33:18 +02:00
Eugenio Lacuesta 890f884de4
Allow 'callback' key in keyword arguments for request callbacks (#5251) 2021-10-01 09:50:42 +02:00
Andrey Rahmatullin 8284de5e76
Fix/silence the Pylint messages added in 2.10 (#5235) 2021-08-24 12:15:29 +02:00
D R Siddhartha 43ea21e830
Feed post-processing plugin support (#5190) 2021-08-24 11:48:01 +02:00
Matsievskiy S.V 3f635eb683
Extract domain from genspider URL (#4439) 2021-08-24 11:05:50 +02:00
Andrey Rahmatullin ada539a63a
Merge pull request #4676 from Gallaecio/startproject-existing-folder 
Fix startproject failing for existing folders
 2021-08-24 12:18:49 +05:00
Andrey Rahmatullin 731f2d3085
Merge pull request #4178 from elacuesta/remove_spider_make_requests_from_url 2021-08-20 16:16:25 +05:00
Eugenio Lacuesta 5792e1c922
Merge remote-tracking branch 'upstream/master' into case-insensitive-dict 2021-08-19 14:09:39 -03:00
Eugenio Lacuesta bbeed6ae8f
CaseInsensitiveDict: preserve original keys (only lookups are key-insensitive) 2021-08-19 14:09:30 -03:00
Andrey Rakhmatullin 5d4b232de8 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-08-16 20:20:43 +05:00
Andrey Rahmatullin 2814e0e197
Disable builtin middlewares in spider middleware tests. (#5229) 2021-08-16 13:22:01 +02:00
Andrey Rakhmatullin b63369c148 Rename tests/requirements-py3.txt to tests/requirements.txt. 2021-08-11 20:02:45 +05:00
Andrey Rahmatullin 0590c3756c
Merge pull request #5221 from wRAR/3.10-tests 
Fixing tests for upcoming 3.10.
 2021-08-11 14:19:51 +05:00
Andrey Rakhmatullin 983b89ad4f Fix SpiderLoaderTest on Python 3.10. 2021-08-11 10:39:23 +05:00
Michel Ace ce9d6c658b
Add more rel_has_nofollow tests 2021-08-10 22:21:51 +02:00
Michel Ace 295f0e2bf5
Make flake8 happy 2021-08-10 21:38:29 +02:00
Michel Ace 07d20a8ce4
Fix test_rel_has_nofollow test 2021-08-10 21:21:43 +02:00
Michel Ace 18b6f30a73
Add test for rel_has_nofollow 2021-08-10 21:13:50 +02:00
Mannan2812 94baa4b272
Fix FileFeedStoragePreFeedOptionsTest fails in CI/CD pipeline (#5198) 2021-08-05 21:23:11 +02:00
Andrey Rakhmatullin 93bf1ae7e3 Fix tests for the 3.10 TypeError message change. 2021-08-03 20:16:59 +05:00
Andrey Rahmatullin b406affd5b
Merge pull request #5210 from laggardkernel/feature/aws-session-token 2021-07-30 15:15:55 +05:00
Eugenio Lacuesta cc89f6be38
Response.attributes (#5218) 2021-07-29 22:12:44 +02:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
Pascal Corpet 70dddfe2b2 Typing: switch to a newer version of MyPy to check types 2021-07-21 17:10:10 +02:00
Andrey Rakhmatullin 7306a81188 Disable builtin middlewares in spider middleware tests. 2021-07-19 20:09:11 +05:00
Adrián Chaves 8a2768561f
Merge branch 'master' into asyncio-parse-asyncgen-proper-rebased 2021-07-16 14:32:15 +02:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Iván de Prado eca641aa3d
Update tests/test_engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:20 +01:00
Iván de Prado Alonso f35970778b Test case for raising CloseSpider on spider idle signal handler 2021-06-29 13:21:38 +01:00
Andrey Rahmatullin 016c7e92d1
Merge pull request #3579 from maramsumanth/duplicate_key 
[MRG+1] Issue #2919: Fix FormRequest.formdata with GET method duplicates same key in query string
 2021-06-15 13:36:53 +05:00
ajaymittur28 7d653288e3 Update unittest 2021-06-14 21:39:18 +05:30
ajaymittur28 857e0e225a Merge branch 'master' into http-proxy-schema 2021-06-14 21:18:49 +05:30
Andrey Rahmatullin 66e2004239
Fix a flake8 problem 2021-06-14 18:58:35 +05:00
Andrey Rahmatullin 16be658a31
Merge branch 'master' into duplicate_key 2021-06-14 18:45:33 +05:00
Andrey Rakhmatullin ecfc924ca8 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-06-04 18:55:32 +05:00
Vostretsov Nikita 23cfdb058e
Reducing amount of warnings during test run (#5162) 
* put flake8 options into separate file to remove pytest warnings

* remove ResourceLeaked warning in pypy

* suppress warnings from twisted

* ignore deprecation warnings here

* ignore deprecation warning in tests of deprecated methods

* ignore deprecation warnings here

* update test classes

* don`t use deprecated method call

* ignore deprecation warnings here

* proper warning class

* more selective ignoring

* Revert "don`t use deprecated method call"

This reverts commit 59216ab560.
 2021-05-28 14:45:06 +05:00
Eugenio Lacuesta 52d0df5f98
CaseInsensitiveDict (deprecate CaselessDict) 2021-05-12 13:08:08 -03:00
Adrián Chaves c5b1ee8101
Make Twisted[http2] installation optional (#5113) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 09:04:53 +02:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Eugenio Lacuesta 02ae1deaf4
Deprecate unused squeues (#5117) 2021-04-27 09:41:44 -03:00
Eugenio Lacuesta ddea6b7bfa
Scheduler: minimal interface, API docs (#3559) 2021-04-26 16:16:14 -03:00
Eugenio Lacuesta 6837919798
Add peek method to queues (#5112) 2021-04-26 14:55:02 -03:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Andrey Rakhmatullin ffc6f525ce Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-04-14 17:34:58 +05:00
Eugenio Lacuesta d8d1dc5b50
Ignore typing warning in test 2021-04-12 10:43:02 -03:00
Eugenio Lacuesta b6f77806b0
Engine tests: fix item class spider, add minimal type hints 2021-04-09 13:09:47 -03:00
Eugenio Lacuesta 91f8144552
Remove deprecated Spider.make_requests_from_url method 2021-04-07 15:16:55 -03:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Andrey Rakhmatullin 30ed7fa349 Some cleanup, make sync middlewares fail earlier. 2021-04-02 22:20:56 +05:00
Eugenio Lacuesta 9e7cbc05ae
Fix type for urlencode's doseq argument 2021-04-01 15:22:51 -03:00
Mikhail Korobov 12f2006b7b
Merge pull request #4799 from GeorgeA92/patch-2 
httpcompression stats added
 2021-04-01 22:47:24 +05:00
Mikhail Korobov f0c8d31193
Merge pull request #4694 from Jgaldos/improve-httpstatus-all-meta 
Improve http status all on http error middleware
 2021-04-01 22:43:45 +05:00
Mikhail Korobov d0e23486b4
Merge pull request #4902 from Gallaecio/retry-request 
Implement a retry request function
 2021-04-01 22:23:56 +05:00
Andrey Rakhmatullin 6803779eb7 Add more tests for _process_iterable_universal. 2021-03-26 22:29:07 +05:00
Andrey Rakhmatullin 7b515a0485 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-03-25 22:13:49 +05:00
Andrey Rakhmatullin f422861ef4 Add more tests for spider middlewares. 2021-03-25 20:49:12 +05:00
Andrey Rakhmatullin a6034f942b Add tests for _AsyncCooperatorAdapter. 2021-03-23 22:47:37 +05:00
Andrey Rakhmatullin c51ec1ae1c Drop process_iterable_helper, add _process_iterable_universal. 2021-03-23 22:47:30 +05:00
Eugenio Lacuesta 72e8cea8af
Avoid exceptions in is_generator_with_return_value (#4935) 2021-03-22 11:51:11 -03:00
Adrián Chaves ec5a7918ec
Include Content-Length in HTTP/1.1 responses (#5057) 2021-03-22 11:25:40 +01:00
Mikhail Korobov 0e7ee1256c
Merge pull request #5036 from dmiwell/urllength-dont-skip-silently 
UrlLengthMiddleware: don't skip silently
 2021-03-20 00:55:33 +05:00
Adrián Chaves 308a58aa27
Update CI to support Twisted 21.2.0 (#5027) 2021-03-19 18:39:44 +01:00
Adrián Chaves 8e73e1dfb5 upper-constraints.txt: restrict botocore further 2021-03-18 23:42:29 +01:00
Adrián Chaves 1d83697990
Merge pull request #4769 from scrapy/http2 
Implement experimental HTTP/2 support
 2021-03-18 22:18:12 +01:00
Dmitriy Pomazunovskiy 9cc4513bd6 simpler stats access 2021-03-15 21:38:03 +06:00
Dmitriy Pomazunovskiy 0f254a6afb Test from_settings 2021-03-12 17:11:50 +06:00
Dmitriy Pomazunovskiy d4b2b61255 Use from_settings for backward compatibility 2021-03-12 16:59:37 +06:00
Dmitriy Pomazunovskiy 6e5ea7924c Log skipped urls by length to INFO, add skipped stats 2021-03-12 11:08:41 +06:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Adrián Chaves 5b2d3e17a3
Merge branch 'master' into http2 2021-03-09 14:59:17 +01:00
Eugenio Lacuesta 9e62355271
Allow logger/stats customization in get_retry_request 2021-03-02 12:12:22 -03:00
Adrián Chaves 386e2a51ae tests/test_downloader_handlers_http2.py: fix style issue 2021-02-24 10:41:01 +01:00
Adrián Chaves 12064d799b HTTP/2: improve header handling 2021-02-24 10:37:38 +01:00
Adrián Chaves 5101094207 HTTP/2: test a CONNECT request 2021-02-24 07:33:39 +01:00
Adrián Chaves 7605f19ec4 HTTP/2: test 2 concurrent requests to the same domain 2021-02-23 05:54:48 +01:00
Eugenio Lacuesta 6326178bc5
http2: acceptable protocol update, tests (#4994) 2021-02-22 16:50:51 +01:00
Adrián Chaves 1f7665c4cf Silence a PyLint check on a mistake made for testing purposes 2021-02-22 16:48:10 +01:00
Adrián Chaves 722a33a2ac Fix style issues 2021-02-22 16:42:38 +01:00
Adrián Chaves 80f5003c88 Add tests for get_retry_request 2021-02-22 16:38:38 +01:00
Andrey Rakhmatullin 40eab1d473 Drop a duplicate import. 2021-02-18 19:56:12 +05:00
Andrey Rakhmatullin 58706c6598 Merge branch 'asyncdef-tests' into asyncio-parse-asyncgen-proper-rebased 2021-02-18 17:20:25 +05:00
Andrey Rakhmatullin 49af7c4c8b Drop pytest-twisted, use Scrapy code to install the reactor. 2021-02-18 17:10:30 +05:00
Eugenio Lacuesta e80f37bd3f
Test http2 agent for unsupported scheme 2021-02-17 16:34:29 -03:00
Eugenio Lacuesta ac82a4a8a0
Merge remote-tracking branch 'upstream/master' into http2 2021-02-17 16:33:57 -03:00
Andrey Rakhmatullin abbbfbbb38 Add tests for deferred_f_from_coro_f. 2021-02-12 22:41:16 +05:00
Andrey Rakhmatullin a7557f2a49 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-02-11 16:09:14 +05:00
Andrey Rakhmatullin 54fd371481 Skip uvloop 0.15.0+ on py36. 2021-02-11 14:24:11 +05:00
Adrián Chaves 9ac5b1d021 Adjust test constraints 2021-02-09 09:13:25 +01:00
Adrián Chaves bb72bba178 tox: apply upper constraints to all non-pinned package installations 2021-02-09 09:13:25 +01:00
Adrián Chaves 45345ba6b5 Use constraints.txt to limit pip resolver backtracking 2021-02-09 09:13:25 +01:00
Andrey Rakhmatullin 5cf403295d Remove a duplicate definition. 2021-02-05 19:40:14 +05:00
Andrey Rakhmatullin 2152a2a508 Add main infrastructure for async callbacks. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin d66d52d3ed Add process_iterable_helper. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin 7e9f498e00 Add MutableAsyncChain. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin acff1eb496 Add aiter_errback. 2021-02-05 16:21:56 +05:00
Andrey Rakhmatullin 76abcedaf4 Add as_async_generator. 2021-02-05 16:21:56 +05:00
Adrián Chaves 248800328c Fix test_pinned_twisted_version 2021-02-03 21:13:43 +01:00
Adrián Chaves c8d8b180bf Merge remote-tracking branch 'upstream/master' into http2 2021-02-03 09:37:35 +01:00
Eugenio Lacuesta 80db569aea
Migrate CI to GitHub actions (#4924) 2021-01-01 19:13:39 -03:00
Eugenio Lacuesta e494a3f733
protocol attribute for h2 responses 2020-12-31 11:50:15 -03:00
Eugenio Lacuesta d698b5147b
Merge branch 'master' into http2 2020-12-31 11:13:25 -03:00
Eugenio Lacuesta b83a1a6fbf Disable test under pypy 2020-12-21 17:21:08 +01:00
Adrián Chaves 0567fdc56e Merge remote-tracking branch 'upstream/master' into tox-pip20.2 2020-12-21 11:02:05 +01:00
Adrián Chaves d0af008608
Merge pull request #4878 from elacuesta/response-protocol-attribute 
Response.protocol attribute
 2020-12-16 15:27:40 +01:00
Adrián Chaves f6879c681e SCRAPY_PINNED → _SCRAPY_PINNED 2020-11-23 23:53:03 +01:00
Adrián Chaves 0dc3e6350c Add a test to check the Twisted version in pinned environments 2020-11-23 22:10:45 +01:00
Eugenio Lacuesta 18b05af877
Remove tests/test_utils_http.py 2020-11-23 16:18:58 -03:00
Eugenio Lacuesta 51ca4d0138
Remove deprecated scrapy.utils.gz.is_gzipped function 2020-11-23 15:47:08 -03:00
Adrián Chaves 07379cf9b7
Merge branch 'master' into allow-customizing-export-column-names 2020-11-20 20:06:18 +01:00
Eugenio Lacuesta 2405df49f1
Add tests for Response.protocol=None 2020-11-16 12:50:33 -03:00
joaquin garmendia 85604e1078
Add failed and success count stats to feedstorage backends (#4850) 2020-11-11 21:16:01 +01:00
Eugenio Lacuesta 5e9a99e6a1
Reponse.protocol as string 2020-11-11 11:15:29 -03:00
Eugenio Lacuesta 5b6b56240c
Test Response.protocol attribute 2020-11-11 01:08:37 -03:00
Eugenio Lacuesta c20b34269f
Remove unnecessary pytest-azurepipelines package (#4876) 2020-11-10 15:35:09 -03:00
Mikhail Korobov 91a8451c98
Merge pull request #4874 from stummjr/fix-missing-fstring-prefix-genspider 
Fix incorrect output on the genspider command
 2020-11-10 17:23:28 +05:00
Eugenio Lacuesta b20cfef1e5
Remove unnecessary line from test 2020-11-09 13:58:52 -03:00
Eugenio Lacuesta 7e98a76ac4
Use deferred_from_coro in asyncio test 2020-11-09 12:17:15 -03:00
Valdir Stumm Junior a2c4a7f920 Add missing f-string prefix to genspider output 2020-11-08 20:14:59 -03:00
Adrián Chaves 4b28da4333 Fix syntax error 2020-11-06 16:46:22 +01:00
Adrián Chaves f3064254ce Merge branch 'patch-2' of github.com:GeorgeA92/scrapy into patch-2 2020-11-06 16:42:58 +01:00
Adrián Chaves ee98771fa7 Remove unused variable 2020-11-06 16:42:32 +01:00
Adrián Chaves 1941f607ca Skip 2 additional tests with older Twisted versions 2020-11-06 16:25:56 +01:00
Eugenio Lacuesta 3095d39740
Test: disable asyncio reactor on Windows for Py>=3.8 2020-11-06 12:16:18 -03:00
Adrián Chaves 5d2a9cf5b1
Merge branch 'master' into patch-2 2020-11-06 14:17:54 +01:00
Adrián Chaves a3e53027ec Test HttpCompressionMiddleware subclasses with custom, parameterless __init__ 2020-11-06 14:16:26 +01:00
Adrián Chaves fea5a11899 Also skip test_asyncdef_asyncio on old Twisted versions 2020-11-06 13:00:42 +01:00
Adrián Chaves ea851b910e Clean up Twisted version check 2020-11-06 13:00:31 +01:00
Eugenio Lacuesta 5b5478ae9d
Call asyncio.get_event_loop when installing the asyncio reactor 2020-11-05 14:01:34 -03:00
Adrián Chaves 906626cf0b Skip MiddlewareUsingCoro::test_asyncdef on asyncio and old Twisted 2020-11-04 21:50:12 +01:00
Adrián Chaves 8e7b756727 Solve Flake8-reported issues 2020-11-04 21:26:55 +01:00
Adrián Chaves 7327145bf3 Remove mitmproxy from pinned environments 2020-10-30 21:34:15 +01:00
Georgiy Zatserklianyi a5872a0fad
Fix output file overwrite with -O (FeedExporter updated) (#4859) 2020-10-30 19:36:39 +01:00
Eugenio Lacuesta 585e4a8aee
Replace local server address 2020-10-15 09:07:34 -03:00
GeorgeA92 872b2e4ce4 testcase added for COMPRESSION_ENABLED setting 2020-10-13 18:41:58 +03:00
GeorgeA92 d32d0d2739 testcase added for HttpCompressionMiddleware with no stats 2020-10-13 18:36:41 +03:00
Eugenio Lacuesta 868826b346
CachingHostnameResolver tests 2020-10-09 10:38:07 -03:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header 
Do not process cookies from headers
 2020-10-08 20:20:32 +05:00
Andrey Rahmatullin 45c06cfd80
Merge pull request #4831 from starrify/downloadermw-support-zstd 
Adding support for zstd in HttpCompressionMiddleware
 2020-10-08 14:08:18 +05:00
Adrián Chaves 269fe35d6d
Merge branch 'master' into http2 2020-10-06 21:05:10 +02:00
P. Chen 156bb0a1d4
Fixing the minor typo on test file path in tests/test_downloadermiddleware_httpcompression.py 2020-10-06 19:53:40 +01:00
Adrián Chaves e40788153c Do not consider about: URLs invalid 2020-10-06 19:13:29 +02:00
GeorgeA92 6050604f62 httocompression/response_bytes tests added 2020-10-06 18:59:57 +03:00
Andrey Rahmatullin b1255b016a
Merge pull request #4746 from Gallaecio/fix_iternodes 
Fix iternodes
 2020-10-06 20:14:04 +05:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
P. Chen 6032a9a310
Minor adjustment to the test case in tests/test_downloadermiddleware_httpcompression.py 2020-10-05 23:55:48 +01:00
P. Chen 50e1f35d1f
Adding test cases for the zstd content encoding 2020-10-05 23:43:12 +01:00
D R Siddhartha 0c24cdb257 Improved warning messages a little 2020-10-04 02:09:21 +05:30
Eugenio Lacuesta 797a6690c0
Tests: use classes instead of paths in settings (#4817) 2020-10-01 23:11:11 -03:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Mikhail Korobov f173af6b9d
Merge pull request #4778 from drs-11/unparseable-no-proxy-fix 
Check for unparseable no_proxy values
 2020-10-01 23:57:00 +05:00
Akshay Sharma 159e2b2e2f
allowing to run .pyw files (#4646) 
* allow .pyw in scrapy/commands/runspider.py

* aesthetics

* added tests for '.pyw'

* created class for testing .pyw files

* name=None parameter in get_log

* small fix

* .pyw tests for non-windows

* used @skipIf for tests

* two more tests skipped
 2020-10-01 23:53:08 +05:00
Eugenio Lacuesta 744f352d09
Do not process cookies from headers 2020-10-01 14:52:23 -03:00
Eugenio Lacuesta 894b509d7a
Crawl rule: remove deprecated code 
Remove the compatibility layer that handles 'process_request'
methods that do not receive a 'response' parameter
 2020-09-29 23:37:28 -03:00
Adrián Chaves 6ef7c44061 Fix timezone test issue 2020-09-22 12:45:21 +02:00
GeorgeA92 c22e810658 httocompression tests added 2020-09-22 07:47:37 +03:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00
Adrián Chaves c3b740f078 Use mocking for tests/test_pipeline_files.py::TestS3FilesStore::test_persist 2020-09-21 23:25:37 +02:00
Adrián Chaves 35726da434 tests/test_feedexport.py: remove unused import 2020-09-21 22:55:25 +02:00
Adrián Chaves 17e135377a Use mocking for tests/test_feedexport.py::BatchDeliveriesTest::test_s3_export 2020-09-21 22:54:39 +02:00
Adrián Chaves 56f05fb164 Use mocking for tests/test_feedexport.py::S3FeedStorageTest::test_store 2020-09-21 22:01:09 +02:00
Adrián Chaves 008cf1c75e Remove a test that has never been executed in Python 3 2020-09-21 20:45:21 +02:00
Andrey Rahmatullin 5e997587d9
Remove dead boto2 code, deprecate is_botocore() (#4776) 2020-09-20 15:06:46 +02:00
Eugenio Lacuesta 6e8d20a07a
HTTP/2: add some type hints (#4785) 2020-09-16 09:57:07 +02:00
drs-11 82ba7c8b52 created separate test for invalid no-proxy values 2020-09-10 20:56:39 +05:30
drs-11 959222df7e check for unparseable no_proxy values 2020-09-05 21:32:05 +05:30
drs-11 35e2d25689 Merge remote-tracking branch 'upstream/master' into 3689-update 2020-09-02 20:51:09 +05:30
drs-11 6565adc471 added test case for get_images 2020-09-02 20:44:26 +05:30
Andrey Rahmatullin c1cc3f2f42
Merge pull request #4761 from Gallaecio/on-the-fly-certificates 
Generate localhost keys for tests on the fly
 2020-09-02 13:37:44 +05:00
Adrián Chaves 9731d91f34
Merge branch 'master' into fix_iternodes 2020-09-01 12:38:37 +02:00
Eugenio Lacuesta 307e35c664
Improve check for invalid cookie in CookiesMiddleware (#4772) 2020-09-01 11:04:00 +02:00
Adrián Chaves 8a3ba34a75 Merge remote-tracking branch 'upstream/master' into http2 2020-08-31 16:43:32 +02:00
Aditya eff33a2e79 fix(h2): Mockserver test uses H2DownloadHandler 2020-08-30 23:55:17 +05:30
maranqz 71d2c2f1a3 improve view of dict 2020-08-30 12:43:44 +03:00
maranqz fc3c66ce95 fix tests: 
* FeedExportConfigTestCase.test_feed_complete_default_values_from_settings_empty
 * FeedExportConfigTestCase.test_feed_complete_default_values_from_settings_non_empty
 2020-08-30 11:44:48 +03:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Mikhail Korobov be655b855d
Merge pull request #4765 from scrapy/simplify-test_crawl 
Simplify tests/test_crawl
 2020-08-29 13:31:20 +05:00
Andrey Rakhmatullin 8123c42737 Simplify running spiders in CrawlSpiderTestCase. 2020-08-29 12:01:58 +05:00
Ammar Najjar ffdf6fe100
use f-strings for the newly merged code from master 
Issue: #4324
 2020-08-29 07:21:48 +02:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Mikhail Korobov 7e6476ff0a
Merge pull request #4764 from scrapy/test_crawl-move-py36 
Merge back tests/py36/_test_crawl.py.
 2020-08-29 00:12:53 +05:00
Andrey Rahmatullin 59a0157ef1
Merge pull request #4722 from Gallaecio/umask 
Do not let umask affect the permissions of startproject-generated files
 2020-08-28 18:41:02 +05:00
Andrey Rakhmatullin de640f41ec Merge back tests/py36/_test_crawl.py. 2020-08-28 18:27:36 +05:00
Ammar Najjar 0e57918231
test(Slot): cover __repr__ 
Issue: #4324
 2020-08-28 13:58:32 +02:00
Mikhail Korobov 64905e3397
Merge pull request #4691 from elacuesta/typing-spider-attributes 
Typing: annotate a few Spider attributes
 2020-08-28 00:03:40 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Adrián Chaves dd378b4bb1 Generate localhost keys for tests on the fly 2020-08-27 11:33:18 +02:00
Mikhail Korobov a6b67cf4a1
Merge pull request #4755 from maranqz/csv-item-exporter-errors 
Add errors parameter for CsvItemExporter with tests
 2020-08-27 12:51:53 +05:00
Ammar Najjar 5ab1a318e8
test: list appears in ValueError Exception message 
Issue #4324
 2020-08-26 15:11:46 +02:00
Eugenio Lacuesta a8114d3731
Typing: annotate a few Spider attributes 2020-08-26 09:00:36 -03:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Mikhail Korobov 0ccaf89a6f
Merge pull request #4756 from ivanprado/master 
Support for delegated methods as callbacks
 2020-08-26 00:10:43 +05:00
Adrián Chaves 2f28cee3ce Add a test to cover searching for a missing node name 2020-08-25 17:49:17 +02:00
Adrián Chaves 6eb1fc4ab5 Merge remote-tracking branch 'upstream/master' into fix_iternodes 2020-08-25 17:01:54 +02:00
Andrey Rahmatullin 067125c303
Merge pull request #4752 from drs-11/master 
DOWNLOAD_MAXSIZE logger level changed from Error to Warning
 2020-08-25 18:35:50 +05:00
Adrián Chaves 39affea93c
Fix style issues 2020-08-25 13:57:48 +02:00
maranqz a2d6fa5adc Add errors parameter for CsvItemExporter with tests 2020-08-25 13:34:43 +03:00
Iván de Prado 3e726b9df7 Support for delegated methods as callbacks 
It can be useful to structure the spiders code around some helper classes.
 2020-08-25 11:31:20 +01:00
Andrey Rahmatullin 64e0ea4ee5
Merge pull request #4735 from wRAR/ciphers-tests 
Re-enable TLS 1.2 in cipher tests.
 2020-08-25 11:48:31 +05:00
drs-11 1f9598ada4 Update branch to latest 2020-08-25 02:14:22 +05:30
drs-11 8b84a65a6b cleaned up code relating to issue #3689 2020-08-25 00:30:17 +05:30
WinterComes 2d8ec9d44f Change DOWNLOAD_MAXSIZE logger level from Error to Warning 2020-08-24 20:37:40 +05:30
Aditya 26d344b762 Merge branch 'http2' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-24 14:58:23 +05:30
Eugenio Lacuesta f1250177dc
Remove Python 3.5 from CI (#4743) 2020-08-22 09:33:35 +02:00
Adrián Chaves e90be0d8a5 Mark the new test as xfail for xmliter_lxml 2020-08-21 14:09:52 +02:00
Adrián Chaves d711eca4d9 Merge remote-tracking branch 'upstream/master' into fix_iternodes 2020-08-21 13:52:52 +02:00
Grisha Temchenko d68aab992e
Smarter generator check for combined return/yield statements (#4721) 2020-08-20 10:22:07 -03:00
sakshamb2113 42383cc267
Add a setting to customize the asyncio event loop (#4414) 2020-08-19 09:18:14 +02:00
Aditya a87ab71d10 refactor(http2): metadata for Stream 
- Add Note about HTTP/2 Cleartext not supported in settings.rst
 2020-08-18 05:52:40 +05:30
Andrey Rakhmatullin d9e69bfb51 Re-enable TLS 1.2 in cipher tests. 2020-08-17 19:46:24 +05:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Mikhail Korobov 282a6d4fc1
Merge pull request #4727 from wRAR/mitmproxy-ca 
Use a longer key in mitmproxy-ca.pem.
 2020-08-17 14:06:58 +05:00
Mikhail Korobov 61459de476
Merge pull request #4688 from ashellunts/obsolete-s3feednokeys 
Remove obsolete S3FeedStorage instancing without AWS credentials
 2020-08-17 13:58:32 +05:00
Mikhail Korobov 5ac135f036
Merge pull request #4718 from Gallaecio/support-pinned-3.6 
Upgrade minimum dependencies for Python 3.6 support
 2020-08-17 13:54:45 +05:00
Ajay Mittur a8e08d51cd
Check if file is already present on running `scrapy genspider` and terminate if so (#4623) 2020-08-17 10:45:52 +02:00
Eugenio Lacuesta 2aa4f3cbf9
Conditional request attribute binding for responses (#4632) 2020-08-17 10:39:59 +02:00
adityaa30 75fe3d1365 fix: increase timeout to 0.5 seconds 
- In Windows specifically the reactor was left unclean by the
  HostnameEndpoint due to the tearDown method of
  test_downloader_handlers.py::HttpTestCase due to
  which the following 2 tests were failing:
  1. test_timeout_download_from_spider_server_hangs
  2. test_timeout_download_from_spider_nodata_rcvd
- Increasing the timeout fixed the test (in local)
 2020-08-17 04:18:03 +05:30
Aditya d97cf973dd Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-16 11:31:10 +05:30
Aditya af73f141b2 refactor: move all http2 tests in separate files 2020-08-16 11:26:32 +05:30
Andrey Rakhmatullin 756c368a6b Use a longer key in mitmproxy-ca.pem. 2020-08-14 22:09:24 +05:00
Adrián Chaves 24ba5a71ac Maybe the problem is not in the code after all 2020-08-13 06:35:09 +02:00
Adrián Chaves 5f4df622a1 test_utils_iterators.py: support Windows the right way 2020-08-13 05:41:06 +02:00
Adrián Chaves 4c0afb606c Update permission expectations 2020-08-12 17:45:26 +02:00
Adrián Chaves 125a058340 Do not let umask affect the permissions of startproject-generated files 2020-08-12 17:07:21 +02:00
Adrián Chaves b1de55d37d Fix marker syntax 2020-08-12 12:34:40 +02:00
Adrián Chaves 8e393a0b21 Do not change the mitmproxy version for no-3.6 Python versions 2020-08-12 12:29:51 +02:00
Adrián Chaves 394631fc0a Restore 3.5 support for mitmproxy-based tests 2020-08-12 12:08:09 +02:00
Ajay Mittur 1c4b4cc6b0
Support defining file path based on item in media pipelines (#4686) 2020-08-11 14:12:44 +02:00
Adrián Chaves 5138f9a965 Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-08-11 13:42:05 +02:00
Adrián Chaves aefd43a6c6 Upgrade minimum dependencies for Python 3.6 support 2020-08-11 12:52:54 +02:00
Aditya c67d6dea31 fix: H2 docs, NotImplementedError for H2 Tunnel 2020-08-11 04:39:41 +05:30
Andrey Rahmatullin 0cf1340c29
Merge pull request #4663 from ajaymittur28/scrapy-check-test 
Add Tests to `scrapy check` command
 2020-08-10 16:18:29 +05:00
Aditya e0c3019d90 fix: ScrapyProxyH2Agent 
- add required test cases

BREAKING CHANGES
Presently the tests (in test_downloader_handlers.py)
1. test_download_without_proxy
2. test_download_with_proxy_https_timeout

collide with each other when run together. However, if both of the tests
are ran individually then both pass.
 2020-08-09 16:19:35 +05:30
Jose Galdos 13181ba788 Improve http status all on http error middleware 2020-08-06 20:43:16 -05:00
Marc Hernández 336f19f5cc
Change super syntax (#4707) 2020-08-04 20:42:01 +02:00
Artur Shellunts 5ec66be4a4
Merge branch 'master' into obsolete-s3feednokeys 2020-07-30 22:18:00 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Aditya a3fecaf07f test: fix host-name H2DownloadHandler tests 2020-07-30 15:45:27 +05:30
Mikhail Korobov 5e2d1bd187
Merge pull request #4434 from BroodingKangaroo/ISSUE-4250-add_batch_deliveries 
Feed exports: add batch deliveries
 2020-07-29 23:03:28 +05:00
Aditya e8342996f6 test: H2DownloadHandler 
Following tests are skipped as Content-Length header not matching the
data received is considered as a ProtocolError
- test_download_broken_content_cause_data_loss
- test_download_broken_chunked_content_cause_data_loss
- test_download_broken_content_allow_data_loss
- test_download_broken_chunked_content_allow_data_loss
- test_download_broken_content_allow_data_loss_via_setting
- test_download_broken_chunked_content_allow_data_loss_via_setting

BREAKING CHANGES
The following tests currently fail
- test_content_length_zero_bodyless_post_request_headers
- test_host_header_seted_in_request_headers
- test_download_with_maxsize_very_large_file
 2020-07-29 13:51:01 +05:30
Aditya 92bec38591 feat: MethodNotAllowed405, Content-Length header 
- add tests to check for Content-Length header
- raise MethodNotAllowed405 when remote send 'HTTP/2.0 405 Method Not
Allowed'
 2020-07-29 13:49:14 +05:30
Eugenio Lacuesta 5265853937
Use ItemAdapter.field_names when writing header in CsvItemExporter (#4668) 2020-07-28 11:15:14 +02:00
Aditya 3685e99cca test: http2 connection timeout 2020-07-22 14:47:20 +05:30
Aditya 316620b517 chore: pass spider as argument for request method 
- download_maxsize and download_warnsize can now be extracted from the
spider directly and passed to the stream
- remove `partial` flag from the response as per RFC 7540 - Section
8.1.2.6
 2020-07-22 13:57:49 +05:30
Artur Shellunts 430d22e46e Remove not used import warnings 2020-07-21 23:39:04 +02:00
nyov 2829cd4268 Allow use without credentials 2020-07-21 23:38:02 +02:00
nyov 98e8086d1b Adapt S3FeedStorage testcase 2020-07-21 23:38:02 +02:00
nyov ece4fa6c7c Fix ignored testcase: boto is never installed 2020-07-21 23:38:01 +02:00
Andrey Rahmatullin f3372a3753
Merge pull request #4254 from elacuesta/spider.parse 
Change Scraper API to call internal `_parse` method
 2020-07-21 17:37:54 +05:00
Akshay Sharma de297a3a16
enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 (#4403) 
* changed ie. -> i.e.(spelling error) on lines 667, 763 (issue scrapy#4332)

* updated all text files for issue #4332 (ie. -> i.e.)

* Apply ie. → i.e. in source comments

* ie → e.g.

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* error handling

* error handlingy

* raise ValueError

* added in-built function for version comparison

* recommit changes

* changed check -> parse

* version comparison -> parse_version

* added scrapy/utils/display.py in pytest.ini

* Trigger

* Add simple test for scrapy.utils.display._colorize

* Flake8: E501 for tests/test_utils_display.py

* assertEquals -> assertEqual

* Normal formatter for all platforms

* separate test for windows

* all curses under try block

* added global TestStr

* more test added

* small fix

* covering exceptions

* windows test failing

* Refactor output color handling

* Fix pprint test

* fix flake8

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2020-07-20 17:23:38 +05:00
BroodingKangaroo 3e0492741d Another try to fix test errors on Windows 2020-07-19 00:10:29 +03:00
BroodingKangaroo 86f7ac2f2b Try to fix error at Windows 2020-07-17 18:22:47 +03:00
Artur Shellunts 62a4ede5e9
Remove deprecated classes BaseSgmlLinkExtractor, RegexLinkExtractor and SgmlLinkExtractor (#4356) 2020-07-17 12:40:04 +02:00
Adrián Chaves d29bec60d7
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest) (#4504) 
* Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest)

* Log a detailed error message to discover why MockServer is not working

* Go for all lines!

* Disable tests based on mitmproxy while running on PyPy

* Fix test_get_func_args for PyPy 3.6+

* Make testPayloadDefaultCiphers work regardless of OpenSSL default ciphers

* Crossing fingers…

* Rename: testPayloadDefaultCiphers → testPayloadDisabledCipher

* Test the PyPy version currently documented as the minimum required version

* Fix the PYPY_VERSION tag

* Update the documentation about supported PyPy versions

* Also test the latest 3.5 Python version with PyPy

* Fix the PYPY_VERSION value for the latest 3.5 version

* Use pinned dependencies for asyncio and PyPy tests against oldest supported Python versions

* Fix PyPy installation for the pypy3-pinned Tox environment

* Try installing Cython

* Maybe PyPy requires lxml 3.6.0?

* install.rst: minor clarification

* lxml 4.0.0 is required on PyPy

* Require setuptools 18.5+

* Revert "Require setuptools 18.5+"

This reverts commit 017ec33ac2.

* Maintain lxml as a dependency if setuptools < 18.5 is used
 2020-07-17 02:19:24 +05:00
Mikhail Korobov 9a74a71c63
Merge pull request #4682 from noviluni/remove_python2_reminiscence 
remove python 2 reminiscence in cookies
 2020-07-16 23:58:58 +05:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
Mikhail Korobov 0f2f1acf04
Merge pull request #4458 from scrapy/azure-pipelines 
Set up CI with Azure Pipelines
 2020-07-16 18:09:31 +05:00
Marc 0e0d1ad643 remove python 2 reminiscence in cookies 2020-07-16 14:25:46 +02:00
Júlio César Batista 38496a00b7
Use the itemlaoders library (#4516) 2020-07-15 12:08:36 +02:00
Andrey Rahmatullin eb1bc74417
Merge pull request #4674 from ashellunts/delete_deprecated_htmlparserlinkextractor 
Remove deprecated class HtmlParserLinkExtractor
 2020-07-15 12:51:42 +05:00
Aditya 1dd27a92fa feat: Idle Timeout for H2Connection (240s) 2020-07-14 22:46:32 +05:30
Artur Shellunts ed5247ca4c Remove htmlparser.py from tests/ignore.txt 2020-07-14 18:06:11 +02:00
Adrián Chaves 6e119bd3e3
Merge branch 'master' into azure-pipelines 2020-07-14 11:26:19 +02:00
Aditya aeaeb7385b feat: assert negotiated protocol as h2 
- implement IHandshakeListener in H2ClientProtocol to know when
handshake is completed
- implement IProtocolNegotiationFactory in H2ClientFactory to provide
information about the acceptableProtols (h2) during NPN or ALPN protocol
 2020-07-14 03:55:14 +05:30
Adrián Chaves 544c1f6e39 Fix the issue 2020-07-13 16:30:34 +02:00
Adrián Chaves 0770961054 Write a test for #4665 2020-07-13 16:05:57 +02:00
Adrián Chaves 53c323b19d _is_path → _is_filesystem_path 2020-07-13 15:29:30 +02:00
Adrián Chaves d54c4496ee Refactor guess_scheme 2020-07-13 14:37:08 +02:00
Andrey Rahmatullin 0ebba175ea
Merge pull request #4666 from Gallaecio/execution-access 
Fix permission handling on project generation from template files
 2020-07-13 16:01:06 +05:00
Andrey Rahmatullin b6f8693db9
Merge pull request #4673 from elacuesta/remove-backslash 
Remove backslash
 2020-07-13 11:14:00 +05:00
Aditya 64c6af10e1 refactor: use str instead of to_unicode 2020-07-13 00:57:49 +05:30
Aditya Kumar 3f7e8635f4
Allow the parse command to write data to a file (#4377) 2020-07-11 08:48:24 +02:00
Eugenio Lacuesta 9aea1f0961
Remove backslash (tests) 2020-07-09 11:04:46 -03:00
Eugenio Lacuesta 2be2bdd2df
Merge remote-tracking branch 'upstream/master' into spider.parse 2020-07-08 14:00:54 -03:00
ajaymittur28 75bff7b6d3 Update url contract value 2020-07-08 19:48:42 +05:30
Aditya 2ea7d82534 feat: H2ClientFactory 2020-07-08 18:57:13 +05:30
ajaymittur28 3e98ed24b6 Convert f-string to .format() 2020-07-08 17:13:57 +05:30
Ajay Mittur b6d4f28ea7
Merge branch 'master' into scrapy-check-test 2020-07-08 15:13:37 +05:30
Adrián Chaves 7e38615703 Remove unused import 2020-07-07 15:30:19 +02:00
Adrián Chaves ca77ca1f75 Generate read-only files on the fly 2020-07-07 14:44:03 +02:00