RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,225 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

11225 Commits

Author SHA1 Message Date
Adrián Chaves 6eaf0c5cc9 Use Ubuntu Bionic for PyPy tests to try to get a newer OpenSSL version recognized 2020-11-04 21:54:00 +01:00
Adrián Chaves 906626cf0b Skip MiddlewareUsingCoro::test_asyncdef on asyncio and old Twisted 2020-11-04 21:50:12 +01:00
Adrián Chaves 8e7b756727 Solve Flake8-reported issues 2020-11-04 21:26:55 +01:00
Adrián Chaves 7327145bf3 Remove mitmproxy from pinned environments 2020-10-30 21:34:15 +01:00
Georgiy Zatserklianyi e9c3188189
Update scrapy/downloadermiddlewares/httpcompression.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-10-30 21:23:29 +02:00
Georgiy Zatserklianyi a5872a0fad
Fix output file overwrite with -O (FeedExporter updated) (#4859) 2020-10-30 19:36:39 +01:00
Adrián Chaves 3e5bc77737 Remove pinning from the PyPy job 2020-10-30 10:31:33 +01:00
Adrián Chaves 13bcdc9f88 Restore pinned dependencies in tox.ini 2020-10-30 10:28:42 +01:00
Adrián Chaves 7187247c01 Add PyDispatcher>=2.0.5 back to dependencies for old pip 2020-10-30 10:23:59 +01:00
Andrey Rahmatullin 75f35f558f
Merge pull request #4803 from elacuesta/instantiate-resolution-receiver 
CachingHostnameResolver
 2020-10-20 13:55:07 +05:00
Eugenio Lacuesta 585e4a8aee
Replace local server address 2020-10-15 09:07:34 -03:00
GeorgeA92 872b2e4ce4 testcase added for COMPRESSION_ENABLED setting 2020-10-13 18:41:58 +03:00
GeorgeA92 d32d0d2739 testcase added for HttpCompressionMiddleware with no stats 2020-10-13 18:36:41 +03:00
GeorgeA92 fd663fd4ad __init__ stats parameter - optional, stats==None - covered. 2020-10-13 18:35:06 +03:00
Adrián Chaves c340e72988 Bump version: 2.3.0 → 2.4.0 2020-10-11 22:12:45 +02:00
Adrián Chaves 47eac83740 Set a release date for Scrapy 2.4.0 2020-10-11 22:11:14 +02:00
Adrián Chaves 015c82b974
Scrapy 2.4 release notes (#4808) 2020-10-11 22:09:45 +02:00
Eugenio Lacuesta 868826b346
CachingHostnameResolver tests 2020-10-09 10:38:07 -03:00
Eugenio Lacuesta 8fe5876597
HostResolution implementation 2020-10-09 10:38:07 -03:00
Eugenio Lacuesta b55c911ddc
Fix CachingHostnameResolver 2020-10-09 10:38:03 -03:00
Andrey Rahmatullin da426fb3cf
Merge pull request #4839 from elacuesta/pytest_xfail_strict 
Add xfail_strict=true to pytest.ini
 2020-10-09 00:39:05 +05:00
Eugenio Lacuesta 13ae17aecc
Add xfail_strict=true to pytest.ini 2020-10-08 14:04:52 -03:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header 
Do not process cookies from headers
 2020-10-08 20:20:32 +05:00
Andrey Rahmatullin 45c06cfd80
Merge pull request #4831 from starrify/downloadermw-support-zstd 
Adding support for zstd in HttpCompressionMiddleware
 2020-10-08 14:08:18 +05:00
Andrey Rahmatullin 8fc4e2e011
Merge pull request #4836 from OfirD1/patch-1 
moved the sentence about processing pending requests when a spider is closed onto a generic note.
 2020-10-07 16:00:38 +05:00
Andrey Rahmatullin ded9a5a085
Merge pull request #4835 from Gallaecio/about-url-support 
Do not consider about: URLs invalid
 2020-10-07 14:50:28 +05:00
dswij 9461414b14 minor changes to remove unnecessary lines 2020-10-07 11:26:53 +07:00
Adrián Chaves 269fe35d6d
Merge branch 'master' into http2 2020-10-06 21:05:10 +02:00
OfirD1 1a597d5e3d
moved the sentence about processing pending requests when a spider is closed onto a generic note. 2020-10-06 21:54:42 +03:00
P. Chen 156bb0a1d4
Fixing the minor typo on test file path in tests/test_downloadermiddleware_httpcompression.py 2020-10-06 19:53:40 +01:00
P. Chen 2e734e6b35
Minor update on the import order in scrapy/downloadermiddlewares/httpcompression.py 2020-10-06 19:51:05 +01:00
dswij 9f02df20c5 Remove PyDispatcher from general requirements 2020-10-07 01:10:23 +07:00
Adrián Chaves e40788153c Do not consider about: URLs invalid 2020-10-06 19:13:29 +02:00
GeorgeA92 6050604f62 httocompression/response_bytes tests added 2020-10-06 18:59:57 +03:00
Andrey Rahmatullin b1255b016a
Merge pull request #4746 from Gallaecio/fix_iternodes 
Fix iternodes
 2020-10-06 20:14:04 +05:00
Andrey Rahmatullin 9b1f86b613
Use f-strings 2020-10-06 18:50:55 +05:00
Eugenio Lacuesta 137c8ba6ee
Docs: mention limitation about Cookie header 2020-10-06 10:50:17 -03:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
dswij ce6884d517
Update tox.ini 2020-10-06 19:51:42 +07:00
dswij 371bb80868 Explicitly declare PyDispatcher as dependencies 2020-10-06 19:45:05 +07:00
P. Chen 6032a9a310
Minor adjustment to the test case in tests/test_downloadermiddleware_httpcompression.py 2020-10-05 23:55:48 +01:00
P. Chen 50e1f35d1f
Adding test cases for the zstd content encoding 2020-10-05 23:43:12 +01:00
P. Chen da3171d4f7
Using the `zstandard` package than `zstd` for supporting frames both with and without the content size info 
See also: https://github.com/sergey-dryabzhinsky/python-zstd/issues/53
 2020-10-05 23:18:58 +01:00
P. Chen c6c3f2ce66
Updating the doc entry for the HTTP compress downloader middleware on zstd 2020-10-05 21:10:40 +01:00
P. Chen 892dd9da57
Adding support for zstd in HttpCompressionMiddleware 2020-10-05 21:03:01 +01:00
D R Siddhartha 0c24cdb257 Improved warning messages a little 2020-10-04 02:09:21 +05:30
Adrián Chaves 004b40a719
as soon as → as long as (#4825) 2020-10-02 19:53:55 -03:00
Eugenio Lacuesta 797a6690c0
Tests: use classes instead of paths in settings (#4817) 2020-10-01 23:11:11 -03:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Mikhail Korobov f173af6b9d
Merge pull request #4778 from drs-11/unparseable-no-proxy-fix 
Check for unparseable no_proxy values
 2020-10-01 23:57:00 +05:00