RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,464 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

8464 Commits

Author SHA1 Message Date
Akshay Sharma 159e2b2e2f
allowing to run .pyw files (#4646) 
* allow .pyw in scrapy/commands/runspider.py

* aesthetics

* added tests for '.pyw'

* created class for testing .pyw files

* name=None parameter in get_log

* small fix

* .pyw tests for non-windows

* used @skipIf for tests

* two more tests skipped
 2020-10-01 23:53:08 +05:00
Habeeb Shopeju f47b120e2b
Documentation of link extractor usage (#4775) 
* Added description when using link extractor outside crawlspiders and created reference documentation for scrapy.link.Link class

* Added link.rst to toctree

* Corrected spelling errors, moved docs to Link doctstring to use autoclass

* Moved link docs to link_extractors

* Update docs/topics/link-extractors.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update link.py

Improvements to URL description

* Update link.py

* Update link.py

Fixed line length Flake issue

* Update link.py

Fixed trailing whitespace

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-10-01 23:50:11 +05:00
Mikhail Korobov 7b1bc9b7fe
Merge pull request #4816 from sashreek1/master 
removed datatype specification for *args & ** kwargs
 2020-10-01 23:46:14 +05:00
Mikhail Korobov 872f68a989
Merge pull request #4818 from elacuesta/crawl-rule-remove-deprecated-code 
Crawl rule: remove deprecated code
 2020-10-01 23:45:14 +05:00
Andrey Rahmatullin e42d82526a
Drop the conditional code for old Twisted (#4820) 2020-10-01 20:22:17 +02:00
Andrey Rahmatullin 44f0fde905
Simplify TLS logging for the modern pyOpenSSL. (#4822) 2020-10-01 20:21:09 +02:00
Sashreek Shankar 9661a8dcfc
removed datatype specification for *args & ** kwargs 2020-10-01 06:46:12 +05:30
Eugenio Lacuesta 894b509d7a
Crawl rule: remove deprecated code 
Remove the compatibility layer that handles 'process_request'
methods that do not receive a 'response' parameter
 2020-09-29 23:37:28 -03:00
Andrey Rahmatullin 5a38639359
Merge pull request #4809 from madeny/patch-1 
Correct some typos
 2020-09-28 18:26:43 +05:00
madeny eff96038c7
Correct some typos 
This won't be an issue if **your** spider doesn't rely on cookies.
 2020-09-26 22:50:38 +02:00
Mirwaisse Djanbaz 3989f64baa
dependecies → dependencies (#4801) 2020-09-21 14:40:00 +02:00
Mirwaisse Djanbaz 7f1e74daa2
dependencides → dependencies (#4800) 2020-09-21 14:38:16 +02:00
Andrey Rahmatullin 5e997587d9
Remove dead boto2 code, deprecate is_botocore() (#4776) 2020-09-20 15:06:46 +02:00
Mikhail Korobov 85e13aff74
Merge pull request #4782 from sripriyesha/patch-1 
add mention of FTP server storage in media storage intro
 2020-09-15 01:17:46 +05:00
KAILASA's Sri Nithya Priyeshananda 7a83474cc5
add mention of FTP server storage in media storage intro 
At the beginning of this doc, in "Specifying where to store the media" feature details, FTP server storage mention was missing
 2020-09-08 17:16:31 +02:00
Andrey Rahmatullin c1cc3f2f42
Merge pull request #4761 from Gallaecio/on-the-fly-certificates 
Generate localhost keys for tests on the fly
 2020-09-02 13:37:44 +05:00
Eugenio Lacuesta 307e35c664
Improve check for invalid cookie in CookiesMiddleware (#4772) 2020-09-01 11:04:00 +02:00
Mikhail Korobov be655b855d
Merge pull request #4765 from scrapy/simplify-test_crawl 
Simplify tests/test_crawl
 2020-08-29 13:31:20 +05:00
Mikhail Korobov 5e9cc3298b
Merge pull request #4324 from ammarnajjar/4307-use-f-strings 
refactor: use f-strings
 2020-08-29 13:28:38 +05:00
Ammar Najjar 90ca9350f5
Update scrapy/commands/check.py 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-29 08:03:03 +00:00
Andrey Rakhmatullin 8123c42737 Simplify running spiders in CrawlSpiderTestCase. 2020-08-29 12:01:58 +05:00
Ammar Najjar ffdf6fe100
use f-strings for the newly merged code from master 
Issue: #4324
 2020-08-29 07:21:48 +02:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Mikhail Korobov 7e6476ff0a
Merge pull request #4764 from scrapy/test_crawl-move-py36 
Merge back tests/py36/_test_crawl.py.
 2020-08-29 00:12:53 +05:00
Andrey Rahmatullin 59a0157ef1
Merge pull request #4722 from Gallaecio/umask 
Do not let umask affect the permissions of startproject-generated files
 2020-08-28 18:41:02 +05:00
Andrey Rakhmatullin de640f41ec Merge back tests/py36/_test_crawl.py. 2020-08-28 18:27:36 +05:00
Ammar Najjar 0e57918231
test(Slot): cover __repr__ 
Issue: #4324
 2020-08-28 13:58:32 +02:00
Mikhail Korobov 64905e3397
Merge pull request #4691 from elacuesta/typing-spider-attributes 
Typing: annotate a few Spider attributes
 2020-08-28 00:03:40 +05:00
Adrián Chaves 3f0a677c04
Cover version directive usage in the documentation policy (#4310) 
* Cover version directives in the documentation policy

* Remove version directives in preparation for Scrapy 2.0

* Update the policy based on the deprecation policy

* Only remove version directives after 3 years
 2020-08-27 23:56:58 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Adrián Chaves dd378b4bb1 Generate localhost keys for tests on the fly 2020-08-27 11:33:18 +02:00
Mikhail Korobov a6b67cf4a1
Merge pull request #4755 from maranqz/csv-item-exporter-errors 
Add errors parameter for CsvItemExporter with tests
 2020-08-27 12:51:53 +05:00
Ammar Najjar 5ab1a318e8
test: list appears in ValueError Exception message 
Issue #4324
 2020-08-26 15:11:46 +02:00
Eugenio Lacuesta a8114d3731
Typing: annotate a few Spider attributes 2020-08-26 09:00:36 -03:00
Ammar Najjar 2ca8dfb4b0
revert f-string changes for files under sep/ 
Issue #4324
 2020-08-26 13:49:39 +02:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
Ammar Najjar 9aaddcde0a
Update scrapy/utils/conf.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:44:20 +00:00
Ammar Najjar ea03e4254f
Update scrapy/http/request/form.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:43:52 +00:00
Ammar Najjar 92dfa7176d
Update scrapy/extensions/statsmailer.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:43:40 +00:00
Ammar Najjar c77450990d
Update scrapy/commands/version.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:41:24 +00:00
maranqz 29725e4b58 Fix tabulation of rst and change documentation link. 2020-08-26 14:38:37 +03:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
maranqz 560c335c07 Add errors parameter in documentation. 2020-08-26 14:00:51 +03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Mikhail Korobov 0ccaf89a6f
Merge pull request #4756 from ivanprado/master 
Support for delegated methods as callbacks
 2020-08-26 00:10:43 +05:00
Iván de Prado 0524df8669 Code simplification. 
Thanks @victor-torres for the suggestion
 2020-08-25 14:37:10 +01:00
Andrey Rahmatullin 067125c303
Merge pull request #4752 from drs-11/master 
DOWNLOAD_MAXSIZE logger level changed from Error to Warning
 2020-08-25 18:35:50 +05:00
Adrián Chaves 39affea93c
Fix style issues 2020-08-25 13:57:48 +02:00
maranqz a2d6fa5adc Add errors parameter for CsvItemExporter with tests 2020-08-25 13:34:43 +03:00
Iván de Prado 3e726b9df7 Support for delegated methods as callbacks 
It can be useful to structure the spiders code around some helper classes.
 2020-08-25 11:31:20 +01:00