RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,531 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

8531 Commits

Author SHA1 Message Date
Adrián Chaves 63becd1bc8
Update news.rst 2020-11-17 21:58:08 +01:00
Adrián Chaves 26836c4e1a Bump version: 2.4.0 → 2.4.1 2020-11-17 09:17:39 +01:00
Adrián Chaves 15d301e968
Cover Scrapy 2.4.1 in the release notes (#4884) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-11-17 09:16:08 +01:00
Eugenio Lacuesta c20b34269f
Remove unnecessary pytest-azurepipelines package (#4876) 2020-11-10 15:35:09 -03:00
Mikhail Korobov 91a8451c98
Merge pull request #4874 from stummjr/fix-missing-fstring-prefix-genspider 
Fix incorrect output on the genspider command
 2020-11-10 17:23:28 +05:00
Mikhail Korobov 27b07c692c
Merge pull request #4872 from elacuesta/asyncio_get_event_loop 
Call asyncio.get_event_loop when installing the asyncio reactor
 2020-11-10 17:22:20 +05:00
Eugenio Lacuesta b20cfef1e5
Remove unnecessary line from test 2020-11-09 13:58:52 -03:00
Eugenio Lacuesta 7e98a76ac4
Use deferred_from_coro in asyncio test 2020-11-09 12:17:15 -03:00
Valdir Stumm Junior a2c4a7f920 Add missing f-string prefix to genspider output 2020-11-08 20:14:59 -03:00
Eugenio Lacuesta 114229eb4a
Docs: add a note about asyncio.set_event_loop 2020-11-06 13:29:14 -03:00
Eugenio Lacuesta 3095d39740
Test: disable asyncio reactor on Windows for Py>=3.8 2020-11-06 12:16:18 -03:00
Eugenio Lacuesta 5b5478ae9d
Call asyncio.get_event_loop when installing the asyncio reactor 2020-11-05 14:01:34 -03:00
Eugenio Lacuesta c292957cb1
Run Windows tests on GitHub actions (#4869) 2020-11-05 15:15:58 +01:00
Georgiy Zatserklianyi a5872a0fad
Fix output file overwrite with -O (FeedExporter updated) (#4859) 2020-10-30 19:36:39 +01:00
Andrey Rahmatullin 75f35f558f
Merge pull request #4803 from elacuesta/instantiate-resolution-receiver 
CachingHostnameResolver
 2020-10-20 13:55:07 +05:00
Eugenio Lacuesta 585e4a8aee
Replace local server address 2020-10-15 09:07:34 -03:00
Adrián Chaves c340e72988 Bump version: 2.3.0 → 2.4.0 2020-10-11 22:12:45 +02:00
Adrián Chaves 47eac83740 Set a release date for Scrapy 2.4.0 2020-10-11 22:11:14 +02:00
Adrián Chaves 015c82b974
Scrapy 2.4 release notes (#4808) 2020-10-11 22:09:45 +02:00
Eugenio Lacuesta 868826b346
CachingHostnameResolver tests 2020-10-09 10:38:07 -03:00
Eugenio Lacuesta 8fe5876597
HostResolution implementation 2020-10-09 10:38:07 -03:00
Eugenio Lacuesta b55c911ddc
Fix CachingHostnameResolver 2020-10-09 10:38:03 -03:00
Andrey Rahmatullin da426fb3cf
Merge pull request #4839 from elacuesta/pytest_xfail_strict 
Add xfail_strict=true to pytest.ini
 2020-10-09 00:39:05 +05:00
Eugenio Lacuesta 13ae17aecc
Add xfail_strict=true to pytest.ini 2020-10-08 14:04:52 -03:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header 
Do not process cookies from headers
 2020-10-08 20:20:32 +05:00
Andrey Rahmatullin 45c06cfd80
Merge pull request #4831 from starrify/downloadermw-support-zstd 
Adding support for zstd in HttpCompressionMiddleware
 2020-10-08 14:08:18 +05:00
Andrey Rahmatullin 8fc4e2e011
Merge pull request #4836 from OfirD1/patch-1 
moved the sentence about processing pending requests when a spider is closed onto a generic note.
 2020-10-07 16:00:38 +05:00
Andrey Rahmatullin ded9a5a085
Merge pull request #4835 from Gallaecio/about-url-support 
Do not consider about: URLs invalid
 2020-10-07 14:50:28 +05:00
OfirD1 1a597d5e3d
moved the sentence about processing pending requests when a spider is closed onto a generic note. 2020-10-06 21:54:42 +03:00
P. Chen 156bb0a1d4
Fixing the minor typo on test file path in tests/test_downloadermiddleware_httpcompression.py 2020-10-06 19:53:40 +01:00
P. Chen 2e734e6b35
Minor update on the import order in scrapy/downloadermiddlewares/httpcompression.py 2020-10-06 19:51:05 +01:00
Adrián Chaves e40788153c Do not consider about: URLs invalid 2020-10-06 19:13:29 +02:00
Andrey Rahmatullin b1255b016a
Merge pull request #4746 from Gallaecio/fix_iternodes 
Fix iternodes
 2020-10-06 20:14:04 +05:00
Andrey Rahmatullin 9b1f86b613
Use f-strings 2020-10-06 18:50:55 +05:00
Eugenio Lacuesta 137c8ba6ee
Docs: mention limitation about Cookie header 2020-10-06 10:50:17 -03:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
P. Chen 6032a9a310
Minor adjustment to the test case in tests/test_downloadermiddleware_httpcompression.py 2020-10-05 23:55:48 +01:00
P. Chen 50e1f35d1f
Adding test cases for the zstd content encoding 2020-10-05 23:43:12 +01:00
P. Chen da3171d4f7
Using the `zstandard` package than `zstd` for supporting frames both with and without the content size info 
See also: https://github.com/sergey-dryabzhinsky/python-zstd/issues/53
 2020-10-05 23:18:58 +01:00
P. Chen c6c3f2ce66
Updating the doc entry for the HTTP compress downloader middleware on zstd 2020-10-05 21:10:40 +01:00
P. Chen 892dd9da57
Adding support for zstd in HttpCompressionMiddleware 2020-10-05 21:03:01 +01:00
Adrián Chaves 004b40a719
as soon as → as long as (#4825) 2020-10-02 19:53:55 -03:00
Eugenio Lacuesta 797a6690c0
Tests: use classes instead of paths in settings (#4817) 2020-10-01 23:11:11 -03:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Mikhail Korobov f173af6b9d
Merge pull request #4778 from drs-11/unparseable-no-proxy-fix 
Check for unparseable no_proxy values
 2020-10-01 23:57:00 +05:00
Akshay Sharma 159e2b2e2f
allowing to run .pyw files (#4646) 
* allow .pyw in scrapy/commands/runspider.py

* aesthetics

* added tests for '.pyw'

* created class for testing .pyw files

* name=None parameter in get_log

* small fix

* .pyw tests for non-windows

* used @skipIf for tests

* two more tests skipped
 2020-10-01 23:53:08 +05:00
Habeeb Shopeju f47b120e2b
Documentation of link extractor usage (#4775) 
* Added description when using link extractor outside crawlspiders and created reference documentation for scrapy.link.Link class

* Added link.rst to toctree

* Corrected spelling errors, moved docs to Link doctstring to use autoclass

* Moved link docs to link_extractors

* Update docs/topics/link-extractors.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update link.py

Improvements to URL description

* Update link.py

* Update link.py

Fixed line length Flake issue

* Update link.py

Fixed trailing whitespace

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-10-01 23:50:11 +05:00
Mikhail Korobov 7b1bc9b7fe
Merge pull request #4816 from sashreek1/master 
removed datatype specification for *args & ** kwargs
 2020-10-01 23:46:14 +05:00
Mikhail Korobov 872f68a989
Merge pull request #4818 from elacuesta/crawl-rule-remove-deprecated-code 
Crawl rule: remove deprecated code
 2020-10-01 23:45:14 +05:00
Andrey Rahmatullin e42d82526a
Drop the conditional code for old Twisted (#4820) 2020-10-01 20:22:17 +02:00