RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,537 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

5053 Commits

Author SHA1 Message Date
Eugenio Lacuesta b0368228d7
Add exception to catch 2020-11-11 11:18:03 -03:00
Eugenio Lacuesta 5e9a99e6a1
Reponse.protocol as string 2020-11-11 11:15:29 -03:00
Eugenio Lacuesta 0fb7bcb2cf
Style adjustment 2020-11-11 01:26:08 -03:00
Eugenio Lacuesta 99cc853d69
Response.protocol attribute 2020-11-11 00:44:11 -03:00
Mikhail Korobov 91a8451c98
Merge pull request #4874 from stummjr/fix-missing-fstring-prefix-genspider 
Fix incorrect output on the genspider command
 2020-11-10 17:23:28 +05:00
Valdir Stumm Junior a2c4a7f920 Add missing f-string prefix to genspider output 2020-11-08 20:14:59 -03:00
Eugenio Lacuesta 5b5478ae9d
Call asyncio.get_event_loop when installing the asyncio reactor 2020-11-05 14:01:34 -03:00
Georgiy Zatserklianyi a5872a0fad
Fix output file overwrite with -O (FeedExporter updated) (#4859) 2020-10-30 19:36:39 +01:00
Andrey Rahmatullin 75f35f558f
Merge pull request #4803 from elacuesta/instantiate-resolution-receiver 
CachingHostnameResolver
 2020-10-20 13:55:07 +05:00
Adrián Chaves c340e72988 Bump version: 2.3.0 → 2.4.0 2020-10-11 22:12:45 +02:00
Eugenio Lacuesta 8fe5876597
HostResolution implementation 2020-10-09 10:38:07 -03:00
Eugenio Lacuesta b55c911ddc
Fix CachingHostnameResolver 2020-10-09 10:38:03 -03:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header 
Do not process cookies from headers
 2020-10-08 20:20:32 +05:00
Andrey Rahmatullin 45c06cfd80
Merge pull request #4831 from starrify/downloadermw-support-zstd 
Adding support for zstd in HttpCompressionMiddleware
 2020-10-08 14:08:18 +05:00
P. Chen 2e734e6b35
Minor update on the import order in scrapy/downloadermiddlewares/httpcompression.py 2020-10-06 19:51:05 +01:00
Adrián Chaves e40788153c Do not consider about: URLs invalid 2020-10-06 19:13:29 +02:00
Andrey Rahmatullin b1255b016a
Merge pull request #4746 from Gallaecio/fix_iternodes 
Fix iternodes
 2020-10-06 20:14:04 +05:00
Andrey Rahmatullin 9b1f86b613
Use f-strings 2020-10-06 18:50:55 +05:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
P. Chen da3171d4f7
Using the `zstandard` package than `zstd` for supporting frames both with and without the content size info 
See also: https://github.com/sergey-dryabzhinsky/python-zstd/issues/53
 2020-10-05 23:18:58 +01:00
P. Chen 892dd9da57
Adding support for zstd in HttpCompressionMiddleware 2020-10-05 21:03:01 +01:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Mikhail Korobov f173af6b9d
Merge pull request #4778 from drs-11/unparseable-no-proxy-fix 
Check for unparseable no_proxy values
 2020-10-01 23:57:00 +05:00
Akshay Sharma 159e2b2e2f
allowing to run .pyw files (#4646) 
* allow .pyw in scrapy/commands/runspider.py

* aesthetics

* added tests for '.pyw'

* created class for testing .pyw files

* name=None parameter in get_log

* small fix

* .pyw tests for non-windows

* used @skipIf for tests

* two more tests skipped
 2020-10-01 23:53:08 +05:00
Habeeb Shopeju f47b120e2b
Documentation of link extractor usage (#4775) 
* Added description when using link extractor outside crawlspiders and created reference documentation for scrapy.link.Link class

* Added link.rst to toctree

* Corrected spelling errors, moved docs to Link doctstring to use autoclass

* Moved link docs to link_extractors

* Update docs/topics/link-extractors.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update link.py

Improvements to URL description

* Update link.py

* Update link.py

Fixed line length Flake issue

* Update link.py

Fixed trailing whitespace

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-10-01 23:50:11 +05:00
Mikhail Korobov 7b1bc9b7fe
Merge pull request #4816 from sashreek1/master 
removed datatype specification for *args & ** kwargs
 2020-10-01 23:46:14 +05:00
Mikhail Korobov 872f68a989
Merge pull request #4818 from elacuesta/crawl-rule-remove-deprecated-code 
Crawl rule: remove deprecated code
 2020-10-01 23:45:14 +05:00
Andrey Rahmatullin e42d82526a
Drop the conditional code for old Twisted (#4820) 2020-10-01 20:22:17 +02:00
Andrey Rahmatullin 44f0fde905
Simplify TLS logging for the modern pyOpenSSL. (#4822) 2020-10-01 20:21:09 +02:00
Eugenio Lacuesta 744f352d09
Do not process cookies from headers 2020-10-01 14:52:23 -03:00
Sashreek Shankar 9661a8dcfc
removed datatype specification for *args & ** kwargs 2020-10-01 06:46:12 +05:30
Eugenio Lacuesta 894b509d7a
Crawl rule: remove deprecated code 
Remove the compatibility layer that handles 'process_request'
methods that do not receive a 'response' parameter
 2020-09-29 23:37:28 -03:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00
Adrián Chaves c3b740f078 Use mocking for tests/test_pipeline_files.py::TestS3FilesStore::test_persist 2020-09-21 23:25:37 +02:00
Andrey Rahmatullin 5e997587d9
Remove dead boto2 code, deprecate is_botocore() (#4776) 2020-09-20 15:06:46 +02:00
drs-11 959222df7e check for unparseable no_proxy values 2020-09-05 21:32:05 +05:30
Adrián Chaves 9731d91f34
Merge branch 'master' into fix_iternodes 2020-09-01 12:38:37 +02:00
Eugenio Lacuesta 307e35c664
Improve check for invalid cookie in CookiesMiddleware (#4772) 2020-09-01 11:04:00 +02:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Ammar Najjar 90ca9350f5
Update scrapy/commands/check.py 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-29 08:03:03 +00:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Andrey Rahmatullin 59a0157ef1
Merge pull request #4722 from Gallaecio/umask 
Do not let umask affect the permissions of startproject-generated files
 2020-08-28 18:41:02 +05:00
Ammar Najjar 0e57918231
test(Slot): cover __repr__ 
Issue: #4324
 2020-08-28 13:58:32 +02:00
Mikhail Korobov 64905e3397
Merge pull request #4691 from elacuesta/typing-spider-attributes 
Typing: annotate a few Spider attributes
 2020-08-28 00:03:40 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Mikhail Korobov a6b67cf4a1
Merge pull request #4755 from maranqz/csv-item-exporter-errors 
Add errors parameter for CsvItemExporter with tests
 2020-08-27 12:51:53 +05:00
Ammar Najjar 5ab1a318e8
test: list appears in ValueError Exception message 
Issue #4324
 2020-08-26 15:11:46 +02:00
Eugenio Lacuesta a8114d3731
Typing: annotate a few Spider attributes 2020-08-26 09:00:36 -03:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
Ammar Najjar 9aaddcde0a
Update scrapy/utils/conf.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:44:20 +00:00