RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,496 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

8496 Commits

Author SHA1 Message Date
Andrey Rahmatullin 8fc4e2e011
Merge pull request #4836 from OfirD1/patch-1 
moved the sentence about processing pending requests when a spider is closed onto a generic note.
 2020-10-07 16:00:38 +05:00
Andrey Rahmatullin ded9a5a085
Merge pull request #4835 from Gallaecio/about-url-support 
Do not consider about: URLs invalid
 2020-10-07 14:50:28 +05:00
OfirD1 1a597d5e3d
moved the sentence about processing pending requests when a spider is closed onto a generic note. 2020-10-06 21:54:42 +03:00
Adrián Chaves e40788153c Do not consider about: URLs invalid 2020-10-06 19:13:29 +02:00
Andrey Rahmatullin b1255b016a
Merge pull request #4746 from Gallaecio/fix_iternodes 
Fix iternodes
 2020-10-06 20:14:04 +05:00
Andrey Rahmatullin 9b1f86b613
Use f-strings 2020-10-06 18:50:55 +05:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
Adrián Chaves 004b40a719
as soon as → as long as (#4825) 2020-10-02 19:53:55 -03:00
Eugenio Lacuesta 797a6690c0
Tests: use classes instead of paths in settings (#4817) 2020-10-01 23:11:11 -03:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Mikhail Korobov f173af6b9d
Merge pull request #4778 from drs-11/unparseable-no-proxy-fix 
Check for unparseable no_proxy values
 2020-10-01 23:57:00 +05:00
Akshay Sharma 159e2b2e2f
allowing to run .pyw files (#4646) 
* allow .pyw in scrapy/commands/runspider.py

* aesthetics

* added tests for '.pyw'

* created class for testing .pyw files

* name=None parameter in get_log

* small fix

* .pyw tests for non-windows

* used @skipIf for tests

* two more tests skipped
 2020-10-01 23:53:08 +05:00
Habeeb Shopeju f47b120e2b
Documentation of link extractor usage (#4775) 
* Added description when using link extractor outside crawlspiders and created reference documentation for scrapy.link.Link class

* Added link.rst to toctree

* Corrected spelling errors, moved docs to Link doctstring to use autoclass

* Moved link docs to link_extractors

* Update docs/topics/link-extractors.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update link.py

Improvements to URL description

* Update link.py

* Update link.py

Fixed line length Flake issue

* Update link.py

Fixed trailing whitespace

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-10-01 23:50:11 +05:00
Mikhail Korobov 7b1bc9b7fe
Merge pull request #4816 from sashreek1/master 
removed datatype specification for *args & ** kwargs
 2020-10-01 23:46:14 +05:00
Mikhail Korobov 872f68a989
Merge pull request #4818 from elacuesta/crawl-rule-remove-deprecated-code 
Crawl rule: remove deprecated code
 2020-10-01 23:45:14 +05:00
Andrey Rahmatullin e42d82526a
Drop the conditional code for old Twisted (#4820) 2020-10-01 20:22:17 +02:00
Andrey Rahmatullin 44f0fde905
Simplify TLS logging for the modern pyOpenSSL. (#4822) 2020-10-01 20:21:09 +02:00
Sashreek Shankar 9661a8dcfc
removed datatype specification for *args & ** kwargs 2020-10-01 06:46:12 +05:30
Eugenio Lacuesta 894b509d7a
Crawl rule: remove deprecated code 
Remove the compatibility layer that handles 'process_request'
methods that do not receive a 'response' parameter
 2020-09-29 23:37:28 -03:00
Andrey Rahmatullin 5a38639359
Merge pull request #4809 from madeny/patch-1 
Correct some typos
 2020-09-28 18:26:43 +05:00
madeny eff96038c7
Correct some typos 
This won't be an issue if **your** spider doesn't rely on cookies.
 2020-09-26 22:50:38 +02:00
Adrián Chaves 6ef7c44061 Fix timezone test issue 2020-09-22 12:45:21 +02:00
Adrián Chaves 07c1d9c25b Remove boto3 as a dependency for tests 2020-09-21 23:32:55 +02:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00
Adrián Chaves c3b740f078 Use mocking for tests/test_pipeline_files.py::TestS3FilesStore::test_persist 2020-09-21 23:25:37 +02:00
Adrián Chaves 35726da434 tests/test_feedexport.py: remove unused import 2020-09-21 22:55:25 +02:00
Adrián Chaves 17e135377a Use mocking for tests/test_feedexport.py::BatchDeliveriesTest::test_s3_export 2020-09-21 22:54:39 +02:00
Adrián Chaves 56f05fb164 Use mocking for tests/test_feedexport.py::S3FeedStorageTest::test_store 2020-09-21 22:01:09 +02:00
Adrián Chaves 008cf1c75e Remove a test that has never been executed in Python 3 2020-09-21 20:45:21 +02:00
Mirwaisse Djanbaz 3989f64baa
dependecies → dependencies (#4801) 2020-09-21 14:40:00 +02:00
Mirwaisse Djanbaz 7f1e74daa2
dependencides → dependencies (#4800) 2020-09-21 14:38:16 +02:00
Andrey Rahmatullin 5e997587d9
Remove dead boto2 code, deprecate is_botocore() (#4776) 2020-09-20 15:06:46 +02:00
Mikhail Korobov 85e13aff74
Merge pull request #4782 from sripriyesha/patch-1 
add mention of FTP server storage in media storage intro
 2020-09-15 01:17:46 +05:00
drs-11 82ba7c8b52 created separate test for invalid no-proxy values 2020-09-10 20:56:39 +05:30
KAILASA's Sri Nithya Priyeshananda 7a83474cc5
add mention of FTP server storage in media storage intro 
At the beginning of this doc, in "Specifying where to store the media" feature details, FTP server storage mention was missing
 2020-09-08 17:16:31 +02:00
drs-11 959222df7e check for unparseable no_proxy values 2020-09-05 21:32:05 +05:30
Andrey Rahmatullin c1cc3f2f42
Merge pull request #4761 from Gallaecio/on-the-fly-certificates 
Generate localhost keys for tests on the fly
 2020-09-02 13:37:44 +05:00
Adrián Chaves 9731d91f34
Merge branch 'master' into fix_iternodes 2020-09-01 12:38:37 +02:00
Eugenio Lacuesta 307e35c664
Improve check for invalid cookie in CookiesMiddleware (#4772) 2020-09-01 11:04:00 +02:00
Ilia Sergunin d10464ca96
Update docs/topics/feed-exports.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-09-01 10:13:40 +03:00
maranqz 71d2c2f1a3 improve view of dict 2020-08-30 12:43:44 +03:00
maranqz fc3c66ce95 fix tests: 
* FeedExportConfigTestCase.test_feed_complete_default_values_from_settings_empty
 * FeedExportConfigTestCase.test_feed_complete_default_values_from_settings_non_empty
 2020-08-30 11:44:48 +03:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Mikhail Korobov be655b855d
Merge pull request #4765 from scrapy/simplify-test_crawl 
Simplify tests/test_crawl
 2020-08-29 13:31:20 +05:00
Mikhail Korobov 5e9cc3298b
Merge pull request #4324 from ammarnajjar/4307-use-f-strings 
refactor: use f-strings
 2020-08-29 13:28:38 +05:00
Ammar Najjar 90ca9350f5
Update scrapy/commands/check.py 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-29 08:03:03 +00:00
Andrey Rakhmatullin 8123c42737 Simplify running spiders in CrawlSpiderTestCase. 2020-08-29 12:01:58 +05:00
Ammar Najjar ffdf6fe100
use f-strings for the newly merged code from master 
Issue: #4324
 2020-08-29 07:21:48 +02:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Mikhail Korobov 7e6476ff0a
Merge pull request #4764 from scrapy/test_crawl-move-py36 
Merge back tests/py36/_test_crawl.py.
 2020-08-29 00:12:53 +05:00