RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
Adrián Chaves 47eac83740 Set a release date for Scrapy 2.4.0 2020-10-11 22:11:14 +02:00
Adrián Chaves 015c82b974
Scrapy 2.4 release notes (#4808) 2020-10-11 22:09:45 +02:00
Eugenio Lacuesta 868826b346
CachingHostnameResolver tests 2020-10-09 10:38:07 -03:00
Eugenio Lacuesta 8fe5876597
HostResolution implementation 2020-10-09 10:38:07 -03:00
Eugenio Lacuesta b55c911ddc
Fix CachingHostnameResolver 2020-10-09 10:38:03 -03:00
Andrey Rahmatullin da426fb3cf
Merge pull request #4839 from elacuesta/pytest_xfail_strict 
Add xfail_strict=true to pytest.ini
 2020-10-09 00:39:05 +05:00
Eugenio Lacuesta 13ae17aecc
Add xfail_strict=true to pytest.ini 2020-10-08 14:04:52 -03:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header 
Do not process cookies from headers
 2020-10-08 20:20:32 +05:00
Andrey Rahmatullin 45c06cfd80
Merge pull request #4831 from starrify/downloadermw-support-zstd 
Adding support for zstd in HttpCompressionMiddleware
 2020-10-08 14:08:18 +05:00
Andrey Rahmatullin 8fc4e2e011
Merge pull request #4836 from OfirD1/patch-1 
moved the sentence about processing pending requests when a spider is closed onto a generic note.
 2020-10-07 16:00:38 +05:00
Andrey Rahmatullin ded9a5a085
Merge pull request #4835 from Gallaecio/about-url-support 
Do not consider about: URLs invalid
 2020-10-07 14:50:28 +05:00
dswij 9461414b14 minor changes to remove unnecessary lines 2020-10-07 11:26:53 +07:00
Adrián Chaves 269fe35d6d
Merge branch 'master' into http2 2020-10-06 21:05:10 +02:00
OfirD1 1a597d5e3d
moved the sentence about processing pending requests when a spider is closed onto a generic note. 2020-10-06 21:54:42 +03:00
P. Chen 156bb0a1d4
Fixing the minor typo on test file path in tests/test_downloadermiddleware_httpcompression.py 2020-10-06 19:53:40 +01:00
P. Chen 2e734e6b35
Minor update on the import order in scrapy/downloadermiddlewares/httpcompression.py 2020-10-06 19:51:05 +01:00
dswij 9f02df20c5 Remove PyDispatcher from general requirements 2020-10-07 01:10:23 +07:00
Adrián Chaves e40788153c Do not consider about: URLs invalid 2020-10-06 19:13:29 +02:00
GeorgeA92 6050604f62 httocompression/response_bytes tests added 2020-10-06 18:59:57 +03:00
Andrey Rahmatullin b1255b016a
Merge pull request #4746 from Gallaecio/fix_iternodes 
Fix iternodes
 2020-10-06 20:14:04 +05:00
Andrey Rahmatullin 9b1f86b613
Use f-strings 2020-10-06 18:50:55 +05:00
Eugenio Lacuesta 137c8ba6ee
Docs: mention limitation about Cookie header 2020-10-06 10:50:17 -03:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
dswij ce6884d517
Update tox.ini 2020-10-06 19:51:42 +07:00
dswij 371bb80868 Explicitly declare PyDispatcher as dependencies 2020-10-06 19:45:05 +07:00
P. Chen 6032a9a310
Minor adjustment to the test case in tests/test_downloadermiddleware_httpcompression.py 2020-10-05 23:55:48 +01:00
P. Chen 50e1f35d1f
Adding test cases for the zstd content encoding 2020-10-05 23:43:12 +01:00
P. Chen da3171d4f7
Using the `zstandard` package than `zstd` for supporting frames both with and without the content size info 
See also: https://github.com/sergey-dryabzhinsky/python-zstd/issues/53
 2020-10-05 23:18:58 +01:00
P. Chen c6c3f2ce66
Updating the doc entry for the HTTP compress downloader middleware on zstd 2020-10-05 21:10:40 +01:00
P. Chen 892dd9da57
Adding support for zstd in HttpCompressionMiddleware 2020-10-05 21:03:01 +01:00
D R Siddhartha 0c24cdb257 Improved warning messages a little 2020-10-04 02:09:21 +05:30
Adrián Chaves 004b40a719
as soon as → as long as (#4825) 2020-10-02 19:53:55 -03:00
Eugenio Lacuesta 797a6690c0
Tests: use classes instead of paths in settings (#4817) 2020-10-01 23:11:11 -03:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Mikhail Korobov f173af6b9d
Merge pull request #4778 from drs-11/unparseable-no-proxy-fix 
Check for unparseable no_proxy values
 2020-10-01 23:57:00 +05:00
Akshay Sharma 159e2b2e2f
allowing to run .pyw files (#4646) 
* allow .pyw in scrapy/commands/runspider.py

* aesthetics

* added tests for '.pyw'

* created class for testing .pyw files

* name=None parameter in get_log

* small fix

* .pyw tests for non-windows

* used @skipIf for tests

* two more tests skipped
 2020-10-01 23:53:08 +05:00
Habeeb Shopeju f47b120e2b
Documentation of link extractor usage (#4775) 
* Added description when using link extractor outside crawlspiders and created reference documentation for scrapy.link.Link class

* Added link.rst to toctree

* Corrected spelling errors, moved docs to Link doctstring to use autoclass

* Moved link docs to link_extractors

* Update docs/topics/link-extractors.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update link.py

Improvements to URL description

* Update link.py

* Update link.py

Fixed line length Flake issue

* Update link.py

Fixed trailing whitespace

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-10-01 23:50:11 +05:00
Mikhail Korobov 7b1bc9b7fe
Merge pull request #4816 from sashreek1/master 
removed datatype specification for *args & ** kwargs
 2020-10-01 23:46:14 +05:00
Mikhail Korobov 872f68a989
Merge pull request #4818 from elacuesta/crawl-rule-remove-deprecated-code 
Crawl rule: remove deprecated code
 2020-10-01 23:45:14 +05:00
Andrey Rahmatullin e42d82526a
Drop the conditional code for old Twisted (#4820) 2020-10-01 20:22:17 +02:00
Andrey Rahmatullin 44f0fde905
Simplify TLS logging for the modern pyOpenSSL. (#4822) 2020-10-01 20:21:09 +02:00
dswij 95b2e94496 fix comment error on tox and re-add cache for travis 2020-10-02 01:05:45 +07:00
dswij cc81f9ed06 add download setting for tox 2020-10-02 00:56:59 +07:00
Eugenio Lacuesta 744f352d09
Do not process cookies from headers 2020-10-01 14:52:23 -03:00
dswij c83a16898f try removing cache in travis to install pip 2020-10-01 20:59:05 +07:00
dswij 392b489a65 travis 2020-10-01 20:32:49 +07:00
dswij f7201b1427 travis and deps 2020-10-01 19:59:23 +07:00
dswij 0ea6ff1136 travis pip version and deps 2020-10-01 16:14:36 +07:00
dswij 66201737a0 fix travis and deps 2020-10-01 15:47:19 +07:00
dswij f4629fe2cc Update travis-pip and tox deps conflict for pip20.2 2020-10-01 14:58:14 +07:00