RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,430 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

1016 Commits

Author SHA1 Message Date
Adrián Chaves ee7bd9d217 Merge remote-tracking branch 'cw9j-q3vf-hrrv/1.8.4-authorization' into 1.8 2024-02-14 14:35:26 +01:00
Adrián Chaves 71b8741e36 Merge remote-tracking branch '7j7m-v7m3-jqm7/1.8-compression-bomb' into 1.8 2024-02-14 14:33:49 +01:00
Adrián Chaves 73e7c0ed01 Backport the latest 2.11 implementation 2024-02-14 14:25:03 +01:00
Adrián Chaves 1c3d289798 Protect against compression bombs 2023-11-23 16:24:22 +01:00
Adrián Chaves ca832a4360 Drop the Authorization header on cross-domain redirect 2023-11-17 14:08:10 +01:00
Adrián Chaves 2023a6bdcb
Fix the CI (#6149) 2023-11-17 10:12:29 +01:00
Adrián Chaves ced7e4649a
Merge pull request from GHSA-9x8m-2xpf-crp3 
* Backport the 2.6.2 security fix

* Force the same Python version for docs as CI in Tox configuration

* Re-enable TLS 1.2 in cipher tests.

* tox.ini: ensure pypy runs pypy2

* Set the release date of Scrapy 1.8.3

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2022-07-25 13:15:17 +02:00
Adrián Chaves d2589c74c5 test_unbounded_response: to_unicode → custom six code 2022-03-01 15:50:19 +01:00
Adrián Chaves efd72b08f9 tests: unicode → to_unicode 2022-03-01 15:14:59 +01:00
Adrián Chaves c5c2e2c7e9 tests: fix cast (str → unicode) 2022-03-01 15:03:36 +01:00
Adrián Chaves 610ce9b96d Fix typo 2022-03-01 14:18:19 +01:00
Adrián Chaves 6c55f76318 Stop mixing keyword arguments and ** in function calls 2022-03-01 13:43:56 +01:00
Adrián Chaves c4b22a5817 Remove further Python 3 syntax 2022-03-01 13:38:00 +01:00
Adrián Chaves c0e745e3fd Remove Python 3 syntax 2022-03-01 13:30:49 +01:00
Adrián Chaves 91706f58b7
Merge pull request from GHSA-mfjm-vh54-3f96 2022-03-01 12:38:19 +01:00
Adrián Chaves 290e2a2726
Merge pull request from GHSA-cjvr-mfj7-j4j8 
* Do not carry over cookies to a different domain on redirect

* Cover the cookie-domain redirect fix in the release notes

* Restore accidentally-removed release notes content

* Fix redirect Cookie handling when the cookie middleware is disabled

* Update the 1.8.2 release date
 2022-03-01 12:26:05 +01:00
Andrey Rakhmatullin b01d69a1bf Add http_auth_domain to HttpAuthMiddleware. 2021-10-05 21:07:45 +02:00
Adrián Chaves 4183925b4b Travis CI → GitHub Actions 2021-10-05 21:04:36 +02:00
Andrey Rahmatullin 18b808b2e9
Merge pull request #4092 from further-reading/master 
Add Python 3.8 official support
 2019-10-29 16:30:58 +05:00
Roy 93e3dc1b82 [test_downloadermiddleware_httpcache.py] Cleaning text 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 16:12:03 +00:00
Roy b73d217de5 [test_downloadermiddleware_httpcache.py] Fixing pytest mark behaviour 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:55:54 +00:00
Roy c51fb959e2 [test_downloadermiddleware_httpcache] Fixing pytest skip behaviour 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:36:54 +00:00
Roy 4432136fff [test_downloadermiddleware_httpcache] Fixing pytest skip behaviour 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:22:21 +00:00
Roy 16bb3ac20d [test_downloadermiddleware_httpcache] Using skipif approach 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 11:24:09 +00:00
elacuesta 7731814cc2 ItemLoader: improve handling of initial item (#4036) 2019-10-28 10:53:53 +01:00
Roy 20ea912513 [test_downloadermiddleware_httpcache] Making xfails more informative 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:52:01 +00:00
Roy 70b2854590 [test_downloadermiddleware_httpcache] Making xfails more informative 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:51:26 +00:00
Roy b3df0a8415 [test_downloadermiddleware_httpcache] Adding xfails to impacted tests following hack fix 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:28:47 +00:00
Roy 11942c436c [test_downloadermiddleware_httpcache] Trying hack to handle systemerror whjen importing leveldb 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:02:13 +00:00
Roy deacd34c8d [test_downloadermiddleware_httpcache] Attempting to add xfail for leveldb related tests 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 17:39:47 +00:00
Roy Healy 4068797558
Update test_downloadermiddleware_httpcache.py 
Adding xfail denoting that leveldb is not supported in 3.8
 2019-10-27 17:02:17 +00:00
Adrián Chaves 1d5c270ce8
Fix dangling file descriptor in FeedExporter when FEED_STORE_EMPTY is False (#4023) 2019-10-22 15:12:52 +02:00
Eugenio Lacuesta e0fabab5cc
Fix TypeError when using DummyStatsCollector 2019-10-01 18:03:22 -03:00
s-sanjay 2c14692e60 remove .keys() to avoid creating a tmp list/keyview obj (#4031) 
Also add --verbose and --nolinks for code coverage
 2019-09-27 09:56:43 +02:00
Mikhail Korobov 31c631f9ee
Merge pull request #3999 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.item
 2019-09-25 16:13:13 +01:00
Adrián Chaves 1236e9e81e Provide complete API documentation coverage of scrapy.item 2019-09-25 13:24:56 +02:00
Mikhail Korobov 1364e92007
Merge pull request #4015 from elacuesta/remove_deprecated_xlib 
Remove deprecated xlib module
 2019-09-20 15:28:07 +01:00
Mikhail Korobov 17c0d72d04
Merge pull request #4016 from elacuesta/rule_default_link_extractor 
Crawling rules: make link extractors optional
 2019-09-20 15:27:17 +01:00
watsta 0b52fa6ca9 LogFormatter: Add the ability to skip log messages (#3987) 2019-09-16 14:12:04 +02:00
OmarFarrag 13735bcf34 Disallow media extensions unregistered with IANA (#3954) 
Co-Authored-By: s-sanjay <sanjay537@gmail.com>
 2019-09-16 14:04:06 +02:00
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Eugenio Lacuesta c5612f387b
Remove deprecated xlib module 2019-09-13 14:32:05 -03:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
Eugenio Lacuesta b92b114633
[test] cb_kwargs contract 2019-08-31 02:44:09 -03:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin 9a8edf2bf1 Tests for setting SSL ciphers. 2019-08-13 16:53:19 +05:00