RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,158 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

1361 Commits

Author SHA1 Message Date
Eugenio Lacuesta b1ddd7bd7b
Refactor test_squeues.py 2020-05-06 13:44:02 -03:00
Eugenio Lacuesta 8643e8d355
Flake8: remove E123 (Closing bracket does not match indentation of opening bracket's line) 2020-05-06 12:26:04 -03:00
Eugenio Lacuesta 63600243e0
Flake8: remove E125 (Continuation line with same indent as next logical line) 
Also remove E401 from pytest.ini - no occurrences in the codebase
 2020-05-06 10:26:52 -03:00
Eugenio Lacuesta fe0c582ee0
Flake8: remove E127 in tests (continuation line over-indented for visual indent) 2020-05-06 09:49:10 -03:00
Adrián Chaves 98835a0d40
Merge branch 'master' into test-ubuntu-bionic 2020-05-06 13:59:01 +02:00
Eugenio Lacuesta 17c0cf64ae
Flake8: remove W504 code (#4525) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-05-06 00:14:48 +02:00
Eugenio Lacuesta 7988c676a9
Update Item docstring, update BaseItem occurrences 2020-05-05 13:11:01 -03:00
Eugenio Lacuesta 622ce86066
Test: make sure scrapy.item.Item does not issue a deprecation warning 2020-05-04 16:22:24 -03:00
Eugenio Lacuesta f75941f79d
Deprecate scrapy.item.BaseItem 2020-05-04 08:53:32 -03:00
Eugenio Lacuesta 4d625afda5
Merge branch 'master' into bytes_received_signal 2020-05-01 16:11:32 -03:00
Eugenio Lacuesta f787b8483c IPv6 test: check for the absence of DNSLookupError 2020-04-28 17:50:06 +02:00
Eugenio Lacuesta efb6f13deb
Remove assertions from production code (#4440) 2020-04-23 12:40:10 +02:00
ilias-ant 773ddf77ba added more tests to cover the (downloaded, uptodate, cached) status functionality 2020-04-19 14:14:17 +03:00
Mikhail Korobov 209ab05315
Merge pull request #4500 from victor-torres/reference-callback 
Serialize requests with callback references as spider attribute
 2020-04-17 18:58:28 +05:00
Victor Torres 1ade3fc723
trying to improve test coverage 2020-04-17 10:34:34 -03:00
Eugenio Lacuesta 7926da4bd0
Merge branch 'master' into flake8-max-line-length 2020-04-16 16:05:09 -03:00
Mikhail Korobov cf39602c30
Merge pull request #4471 from elacuesta/fix-pycodestyle-e731 
Fix pycodestyle E731 ("do not assign a lambda expression, use a def")
 2020-04-16 23:17:31 +05:00
Eugenio Lacuesta c922992277
Tests: Move code inside __main__ block 2020-04-16 11:37:37 -03:00
Eugenio Lacuesta 1f2e2a6006
Merge branch 'master' into response_ip_address 2020-04-16 11:32:37 -03:00
Victor Torres e0921cab66
remove not used code 
This code is not needed anymore because we're getting the
already mangled name when matching func with spider attributes.
 2020-04-16 11:18:59 -03:00
Victor Torres 47a992615a
serialize requests with callback references as spider attribute 
You could define a spider attribute that references a callback method
but if this method has a different name than your spider attribute,
the request serializer is not able to find it on the spider class.

With this commit we're fixing this behavior as we're searching for
callback references in the spider object itself instead of looking
for attributes with the same function's name, that could be different.
 2020-04-15 20:03:58 -03:00
Mikhail Korobov 2d46777d0a
Merge pull request #4489 from elacuesta/remove-returnValue-function 
Remove twisted.internet.defer.returnValue occurrences
 2020-04-15 22:45:52 +05:00
Eugenio Lacuesta 01d73dd3f0
Merge branch 'master' into flake8-max-line-length 2020-04-15 09:24:33 -03:00
Adrián Chaves d323a57381
Merge branch 'master' into fix-pycodestyle-e731 2020-04-15 14:16:15 +02:00
Eugenio Lacuesta 94d7ad76cb
Fix pycodestyle E2XX (whitespace) (#4468) 2020-04-15 14:11:37 +02:00
Eugenio Lacuesta 4023d5db33
Replace _DefGen_Return exception handling 
Handle StopIteration instead
 2020-04-13 06:35:26 -03:00
Eugenio Lacuesta 1bd8f392c9
Initial removal of twisted.internet.defer.returnValue 2020-04-13 06:12:30 -03:00
ilias-ant 83a0cc6cdf Add status to files information 2020-04-12 23:22:17 +03:00
Eugenio Lacuesta 950a52468c
Merge branch 'master' into flake8-max-line-length 2020-04-10 15:24:22 -03:00
Eugenio Lacuesta 2265c1fda9
Merge branch 'master' into fix-pycodestyle-e731 2020-04-10 14:32:30 -03:00
Eugenio Lacuesta c4a5e3f0da
Simplify bytes_received signal 
Remove "source" parameter
 2020-04-06 09:26:13 -03:00
Eugenio Lacuesta e3342669ae
Merge branch 'master' into bytes_received_signal 2020-04-06 09:17:35 -03:00
Eugenio Lacuesta 4270e0a0da
Fix E731: do not assign a lambda expression 2020-04-04 21:54:55 -03:00
Eugenio Lacuesta e2d5d357a7
Fix pycodestyle E502 2020-04-01 14:24:30 -03:00
Eugenio Lacuesta 182394bcec
E501 compliance (tests) 2020-03-22 23:29:30 -03:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
Andrey Rahmatullin ecd3a976d2
Merge pull request #4398 from nyov/remove-crawler-spiders 
Remove deprecated Crawler.spiders property
 2020-03-16 17:21:07 +05:00
Eugenio Lacuesta aa6075e08c
Merge branch 'master' into bytes_received_signal 2020-03-15 18:05:04 -03:00
Adrián Chaves 3f6cdcabce Restrict pytest to versions prior to 5.4 2020-03-13 13:25:53 +01:00
Lukas Anzinger ccc4d88779
Ignore a domain in allowed_domains with port and issue a warning (#4413) 2020-03-12 20:42:14 +01:00
nyov 3b0820d747
Deprecate Spider.make_requests_from_url, part 2 (#4412) 2020-03-12 20:15:49 +01:00
Mikhail Korobov db9198bc20
Merge pull request #4409 from nyov/remove-deprecation-warning 
Remove a 'twisted.test.proto_helpers' deprecation warning
 2020-03-12 18:46:47 +05:00
Eugenio Lacuesta 8d30dc0888
Response.follow_all: return empty generators for empty sequences 2020-03-12 09:36:15 -03:00
Andrey Rahmatullin 388f23c30c
Merge pull request #3858 from elacuesta/multiple_feed_export_formats 
[MRG+1] Support for exporting to multiple feeds in a single crawl
 2020-03-12 14:31:57 +05:00
Eugenio Lacuesta 21f1eddc90
Merge branch 'master' into bytes_received_signal 2020-03-11 21:16:35 -03:00
Eugenio Lacuesta f3bab819ab
Add tests for scrapy.utils.conf.feed_complete_default_values_from_settings 2020-03-11 20:57:31 -03:00
Eugenio Lacuesta ac73bcc79c
Merge branch 'master' into response_ip_address 2020-03-08 22:30:59 -03:00
Lukas Anzinger 9d9dea0d69 Fix handling of None in allowed_domains. 
Nones in allowed_domains ought to be ignored and there are also tests
for that scenario. This commit fixes the handling of None and also the
accompanying tests which are now executed again.
 2020-03-07 20:34:18 +01:00
nyov 915e363db5 Remove a 'twisted.test.proto_helpers' deprecation warning 2020-03-07 18:03:25 +00:00
Eugenio Lacuesta ada37c5409
Export to multiple formats in a single crawl 2020-03-05 09:33:57 -03:00
nyov b1566a6962 Remove deprecated Crawler.spiders property 
Deprecated since 419026615 (2014, Scrapy 0.25)
 2020-03-04 21:23:42 +00:00
Eugenio Lacuesta 3aa5eab993
Merge branch 'master' into response_ip_address 2020-03-03 13:53:43 -03:00
nyov 1b591ff061 Obsolete deprecated settings 
Obsolete REDIRECT_MAX_METAREFRESH_DELAY
  which has been deprecated since Scrapy 0.18

Obsolete LOG_UNSERIALIZABLE_REQUESTS
  which has been deprecated since Scrapy 1.2.0
  and is replaced by SCHEDULER_DEBUG
 2020-03-03 14:03:30 +00:00
Adrián Chaves 6aa0ba4553
Write release notes for Scrapy 2.0.0 (#4329) 2020-03-03 09:11:11 +01:00
sakshamb2113 c411a51f42
Fix random failures from test_fixed_delay in some machines (#4372) 2020-02-28 13:17:02 +01:00
Adrián Chaves 9aae4c0be7 Add tests for envvar setting warnings 2020-02-27 16:31:43 +01:00
gunblues 034e2c31c7
Use a non-zero exit code when a pipeline's open_spider method throws an exception (#4207) 
* fix issue 4175 - Scrapy does not use a non-zero exit code when pipeline's open_spider throws the exception

* remove extra blank lines

* remove redundant code

* remove blank line at end of file

* more suitable naming for response and make if-condition shorter

* avoid error - AttributeError: 'Deferred' object has no attribute 'result'

* use getattr to make code concisely

* add test

* remove useless file

* modify test class name

* remove unneccessary files

* Fix Flake8-reported issue

* fix these items which are suggested by Gallaecio
・Sort those imports at tests/test_cmdline_crawl_with_pipeline/__init__.py
・Remove the unused setUp method.
・Remove comments generated by Scrapy’s project generation tool.
・Remove the [deploy] section from the scrapy.cfg file (I don’t think it’s needed here)
・Remove BOT_NAME and NEWSPIDER_MODULE from settings.py (I think there are not needed either, although I’m less sure about NEWSPIDER_MODULE)

* have to reserve BOT_NAME, SPIDER_MODULES in settings.py

* Remove unneeded empty lines

* Empty __init__.py file with unneeded comments

* Remove an unneeded empty line at the end

* Remove unneeed empty line from __init__.py file

* Update __init__.py

* Update __init__.py

* Update exception.py

* Update normal.py

* Update __init__.py

* Update __init__.py

* fix W391 blank line at end of file

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-26 00:46:05 +05:00
Eugenio Lacuesta 889b471852
Import changes 2020-02-23 18:40:43 -03:00
Eugenio Lacuesta a44942d27d
Merge branch 'master' into response_ip_address 2020-02-23 18:13:52 -03:00
Mikhail Korobov 49480225d7
Merge pull request #4365 from noviluni/fix_flake8_E701_E271 
Fix flake8 E131, E211, E251, E271, E701
 2020-02-23 15:52:52 +05:00
elacuesta 9d983c1b99
Expose certificate for HTTPS responses (#4054) 
* Expose certificate for HTTPS responses

* Fix test (missing inlineCallbacks decorator)

* Note about Response.certificate

* Explicitly cover None as the default value of Response.certificate

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-22 17:20:31 +05:00
Vostretsov Nikita 67ee0b097f
Remove specifics of downstream request queues from scheduler (#3884) 
* move serialization/deserialization logic to downstream queues

* make memory queues conform to common interface

* make ScrapyPriorityQueue conform common interface

* ScrapyPriorityQueue works with disk

* make key as string

* return list instead of dict as earlier

* downloader aware pq works with new interface

* we don`t need these methods anymore

* create directories for files

* remove dummy priority

* remove priority as parameter, let every queue decide for itself

* rename obj to request

* DownloaderAwarePriorityQueue is too thin wrapper around _SlotPriorityQueues, just remove second one

* remove priority as parameter, let every queue decide for itself

* rename argument

* more granular class separation

* python2 compatible

* one more argument for common interface

* more simple downstream queue interface

* single place for easier customization

* rename function

* shorter

* shorter

* use named arguments

* fix typo

* add docstring

* Update scrapy/pqueues.py

Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>

* Update scrapy/pqueues.py

Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>

* 4 spaces indentation

* we ok with existing directories

* remove unused import

* rename method

* remove unused imports

* it has no sense now

* relining

* note about queues

* add value

* Revert "it has no sense now"

This reverts commit b61604275b.

* pep8 E261

* pep8 E303

* pep8 E501

* pep8 E123

* pep8 E123

* use create instance

* remove excessive import

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-02-22 17:02:57 +05:00
Marc Hernandez Cabot 6e8e117aee fix flake E211 2020-02-22 12:32:38 +01:00
Marc Hernandez Cabot 69a8648bef fix E251 2020-02-22 12:32:38 +01:00
Marc Hernandez Cabot 9ad10bb6f7 fix E131 2020-02-22 12:32:38 +01:00
Marc Hernandez Cabot b49ece0b87 fix E701 and E271 flake8 2020-02-22 12:32:38 +01:00
Mikhail Korobov e551bc5170
Merge pull request #4318 from wRAR/asyncio-parse-asyncgen 
Support yield in async def callbacks.
 2020-02-21 23:50:01 +05:00
Marc Hernandez Cabot 6fb85951ce fix E22X flake8 2020-02-21 08:39:14 +01:00
Marc Hernández 91bbc70bc1
fix E30X flake8 (#4355) 2020-02-21 06:05:31 +01:00
Andrey Rahmatullin 88179027de
Merge pull request #4331 from Gallaecio/response-cb-kwargs 
Implement Response.cb_kwargs
 2020-02-19 22:40:14 +05:00
Andrey Rahmatullin f558df2558
Merge pull request #4188 from elacuesta/logformatter-error-formatting 
LogFormatter error formatting
 2020-02-19 19:05:08 +05:00
Andrey Rahmatullin 528b894f28
Merge pull request #4321 from Gallaecio/link-extractor-encoding 
Use safe_url_string in link extraction
 2020-02-19 18:19:21 +05:00
Adrián Chaves 43b43654a1 Add tests for meta and cb_kwargs not being available 2020-02-13 22:39:58 +01:00
Adrián Chaves 5ff9eb90ea Add a test for the copy of cb_kwargs from Request to Response 2020-02-13 22:36:18 +01:00
Adrián Chaves b4958358e8 Update tests to account for link extractors escaping spaces 2020-02-12 19:00:04 +01:00
Mikhail Korobov a6ef065eb5
Merge pull request #4271 from wRAR/asyncio-signals 
async def support for signal handlers that already supported Deferreds
 2020-02-11 02:05:45 +05:00
Andrey Rakhmatullin 1f0f52cbf7 Improve async signal tests. 2020-02-11 01:05:45 +05:00
Andrey Rakhmatullin 36dcf90184 Also test non-default async callbacks. 2020-02-11 01:00:16 +05:00
Abhishek Pratap Singh 4626e90df8
Allow updating flags in follow and follow_all (#4279) 2020-02-10 19:48:31 +01:00
Eugenio Lacuesta 122ce6d6fb
Check bytes are received in order (bytes_received signal) 2020-02-10 10:20:26 -03:00
Adrián Chaves 35723d76c0 Use canonicalize_url in link extraction 2020-02-07 22:59:53 +01:00
Adrián Chaves c2f484de88
Merge branch 'master' into response_ip_address 2020-02-07 21:49:38 +01:00
Adrián Chaves 4c6993f264
Merge branch 'master' into asyncio-parse-asyncgen 2020-02-07 21:34:54 +01:00
Mikhail Korobov 957681bcfa
Merge pull request #4272 from elacuesta/spider-middleware 
Spider middleware: catch spider callback exceptions early
 2020-02-07 23:40:50 +05:00
Mikhail Korobov afbaf9d430
Merge pull request #4303 from whalebot-helmsman/request_left_downloader_signal 
request_left_downloader signal
 2020-02-07 23:33:51 +05:00
Andrey Rakhmatullin 7323780c97 Support yield in async def callbacks. 2020-02-07 21:32:45 +05:00
Andrey Rakhmatullin 31f6c7112f Add a test for an async callbacks that returns requests. 2020-02-07 17:14:52 +05:00
Andrey Rakhmatullin 4a7c7340a0 Merge remote-tracking branch 'origin/master' into asyncio-signals 2020-02-07 16:58:59 +05:00
Eugenio Lacuesta 6c333491a8
Merge branch 'master' into response_ip_address 2020-02-06 18:53:11 -03:00
Lane Shaw 3263441fbc
Update RFPDupeFilter line separator for correct universal newlines mode usage (#4283) 2020-02-06 22:14:40 +01:00
Mikhail Korobov 042e71e2b8
Merge pull request #4311 from Gallaecio/metarefresh-ignore-tags 
Make METAREFRESH_IGNORE_TAGS an empty list by default
 2020-02-06 23:40:45 +05:00
elacuesta 35dafef7f1
Specify Twisted reactor (TWISTED_REACTOR setting) (#4294) 
* Add the ability to install a specific reactor

* Add docs for the TWISTED_REACTOR setting

* Add tests for the TWISTED_REACTOR setting

* Update asyncio reactor test

* Ignore W503 globally

W503 is not PEP8-compliant:
c59c4376ad

* Line length adjustment

* Adjust asyncio reactor tests

* Merge ASYNCIO_ENABLED and TWISTED_REACTOR settings

* More docs about TWISTED_REACTOR

* Fix asyncio reactor test

* Docs: fix title

* Reword docs

* Check the TWISTED_REACTOR setting outside of the installing function

* Remove unrelated change

* Update scrapy/utils/log.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-06 22:42:34 +05:00
Andrey Rakhmatullin 489ffcda51 Add a test for an async item_scraped handler. 2020-02-06 22:40:11 +05:00
Vostretsov Nikita 4be19e443e name signla catcher in accord with signal name 2020-02-06 13:46:23 +00:00
Vostretsov Nikita 4a91a5427d fix typo 2020-02-06 13:44:51 +00:00
Adrián Chaves 576663e5a7 Make METAREFRESH_IGNORE_TAGS an empty list by default 2020-02-06 10:43:20 +01:00
Vostretsov Nikita 3769f75386 pep8 E302 2020-02-05 12:08:08 +00:00
Vostretsov Nikita 9916f6e556 tests for new signal 2020-02-05 11:32:54 +00:00
Eugenio Lacuesta ad70497416
Remove unnecessary parentheses in class definition 2020-02-04 13:30:13 -03:00
Eugenio Lacuesta e0ef8ad2d6
CrawlerRunner test for Response.ip_address 2020-02-03 15:52:15 -03:00