RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 153 MiB
Commit Graph

9291 Commits

Author SHA1 Message Date
Mikhail Korobov 845bae6637
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-27 08:49:19 +00:00
Mikhail Korobov 820adb69c0
Merge pull request #3701 from Gallaecio/documentation-policy-api 
Update the documentation policies: Ask to use docstrings to document …
 2019-03-27 12:25:38 +05:00
Adrián Chaves ce837b0f74 Update the documentation policies: Ask to use docstrings to document API members 2019-03-27 07:09:18 +01:00
Maram Sumanth 2fd8b7c28c [MRG+1] redirect_reasons in Request.meta (#3687) 2019-03-27 00:15:53 +05:00
Maram Sumanth 5f2ad5377e
fixed typo 2019-03-26 16:46:15 +05:30
Adrián Chaves 431f18a9a1 Document FilesPipeline.file_path and ImagesPipeline.file_path 2019-03-26 12:15:19 +01:00
Maram Sumanth ae856e8ba8
corrected tests 2019-03-26 16:21:52 +05:30
Maram Sumanth 213b9eb879
Update test_http_request.py 2019-03-26 15:59:38 +05:30
Maram Sumanth dc8310e292
changed tests 2019-03-26 15:42:58 +05:30
Adrián Chaves ba13de29e1
Merge branch 'master' into feat-685 2019-03-26 10:38:38 +01:00
Vostretsov Nikita 73e4ff5304 report warnings 2019-03-25 13:48:58 +00:00
Vostretsov Nikita 103fdc40c1 Merge remote-tracking branch 'upstream/master' into round-robin-scheduler-tested 2019-03-25 16:16:42 +03:00
Vostretsov Nikita 31b8a6b33a report warnings 2019-03-25 16:08:40 +03:00
Lucy Wang df574de8cc improve tests and fix some lint warnings (#6) 
* refactor downloader-aware test cases

* fix lint

* add doctest for _path_safe

* remove unused code

* better doctest
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 8afffb7234 Tests Cleanup 
add doctest for function
no need in this variables
move common assertion inside function
rename variable
rename variables
rename function
use function
this is not a method of public API
correct name for test
Update docs/topics/settings.rst

Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 989bba6cb3 Revert "new signal" 
This reverts commit 646164fd7d6dd52061804d2df7424cff929bf739.
remove tests
Revert "emit new signal"

This reverts commit fcde0c6880678957a76af6083b6248f430a00fcf.
Revert "documentation for new signal"

This reverts commit 8aeb9f696ece95c16499a96767a7afa3d9c4abf4.
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 443fb98a47 Use downloader directly 
rename variable
remove old write function
remove unused imports
remove old read function
remove unused function
use mock methods
mock downloader
close downloader
add parse method
use new PQ class
create mock downloader
use downloader directly
remove mark/unmark mechanism
 2019-03-25 16:08:40 +03:00
Mikhail Korobov 83eb537645 assorted cleanups: comments, docstrings, etc 
scheduler cleanup

Scheduler no longer converts requests to dicts; PriorityQueue
instances always work with Request instances; converting Requests
to dicts is now Priority Queue responsibility.
minor cleanup
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 3b1db71dac New signal 
update signature
documentation for new signal
utilize new signal
correct signal handler signature
emit new signal
test another signal
new signal
rename test file
faster test
rename test case
tests for signal emitting in bad cases
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 757f53a324 Address Lucy's comments 
add tests to check correctness of slot setermination
unmark requests after downloading
shorter
better exception message
 2019-03-25 16:08:40 +03:00
Mikhail Korobov 90934959d0 actually apply __slots__ suggestion 
[wip] refactoring

* SlotPriorityQueues doesn't care about objects inside, it is now just
  a container for multiple priority queues
* assorted variable renames
* don't inherit DownloaderAwarePriorityQueue from SlotBasedPriorityQueue
* apply @whalebot-helmsman's suggestions for __slots__ and meta issues
more bike-shedding

* remove mutable default arguments
* more verbose variable names
remove unneeded code

* PriorityAsTupleQueue.is_empty does the same as len(self) == 0
* custom PriorityAsTupleQueue.close is not needed after a switch
  to namedtuples
* is_new and is_empty return values are unused
* "url" local variable is unused
PrioritySlot.__str__ shouldn't return unicode in Python 2

also, do some bike-shedding: _pathable -> _path_safe
use namedtuple for PrioritySlot
cleanup: _get_from_request does the same here

Request.meta is always a dict
 2019-03-25 16:08:40 +03:00
Eugenio Lacuesta 1b4385b7e3 Rule.process_request: move deprecation warnings and compiling code, update tests 2019-03-22 19:46:17 -03:00
Eugenio Lacuesta 174ba3cc56 Rule.process_request: update docs 2019-03-22 19:16:18 -03:00
Eugenio Lacuesta 56929e77d9 Rule.process_request: deprecate the use of functions taking only one argument 2019-03-22 18:34:55 -03:00
Eugenio Lacuesta bbf24b7a1c Rule.process_request: use scrapy.utils.python.get_func_args 2019-03-22 18:02:31 -03:00
Mikhail Korobov 8583c033ef
Merge pull request #3694 from noviluni/master 
[MRG+1] Fix numeration
 2019-03-23 00:23:06 +05:00
Adrián Chaves c191a8e70c
Merge pull request #3698 from noviluni/_fix_gitignore 
[MRG+1] remove duplicated entry in gitignore
 2019-03-22 19:45:57 +01:00
Mikhail Korobov 729580dae2
Merge pull request #3671 from Gallaecio/item-deepcopy 
Implement Item.deepcopy()
 2019-03-22 22:56:39 +05:00
Marc Hernández Cabot 9a0fe8bf2d remove duplicated entry in gitignore 2019-03-22 18:56:25 +01:00
Adrián Chaves 7c148fce5a Implement Item.deepcopy() 2019-03-22 17:07:10 +01:00
Konstantin Lopuhin af2b666595
Merge pull request #3505 from kasun/json_request 
[MRG+2] Request subclass for json requests #3504
 2019-03-22 18:25:54 +03:00
Mikhail Korobov 4196d4869b
Merge pull request #3607 from victor-torres/feed-storage-s3-acl 
[MRG+1] add FEED_STORAGE_S3_ACL setting
 2019-03-22 20:05:45 +05:00
Adrián Chaves 72cf190145 Add a FAQ entry about name collisions 2019-03-22 15:37:01 +01:00
Júlio César Batista 2cb4dc3205
Mentioning to use JSON API for ACLs 2019-03-22 09:50:11 -03:00
Vostretsov Nikita 338b78d796 Add documentation 
add section to broad-crawl topic
reword in accord with broad-crawl topic
add documentation for new priority queue
 2019-03-22 15:40:37 +03:00
Vostretsov Nikita 8e8ce301b1 check CONCURRENT_REQUESTS_PER_IP is not set 2019-03-22 15:40:37 +03:00
Vostretsov Nikita 987c2ae4a9 test ip concurrency incompatibility with DAPQ 2019-03-22 15:40:37 +03:00
Vostretsov Nikita 4163a7a1c7 no need for this 2019-03-22 15:40:37 +03:00
Daniel Graña 6ff2574c27 Needs to be installed within tox env 2019-03-22 15:40:36 +03:00
Daniel Graña 7d3175ac84 Fix boto import error under Jessie testing environment 2019-03-22 15:40:36 +03:00
Vostretsov Nikita d970be64cc Integration test 
integration testing only everything is working, not logic of PQ
use method
create slot attribute in constructor
corect class for test case
stop crawler in teardown method
use class
correct entity naming
python 2 adaptation
integration test with crawler and spider
 2019-03-22 15:40:36 +03:00
Vostretsov Nikita a23e1894b3 Fix boto problem 
another way to fix boto problem
Revert "fix for travis ci based on https://github.com/boto/boto/issues/3717"

This reverts commit 150d2564ff0ea994652da7f5be333d72e0b38d93.
fix for travis ci based on https://github.com/boto/boto/issues/3717
 2019-03-22 15:40:36 +03:00
Vostretsov Nikita a46613afa8 use regular comments 2019-03-22 15:40:36 +03:00
Vostretsov Nikita 6af964cc0b common indentation for comment 2019-03-22 15:40:36 +03:00
Vostretsov Nikita 484927b08c less complex implementation 2019-03-22 15:40:36 +03:00
Vostretsov Nikita 0e06b9a816 use urlparse_cached where it is possible 2019-03-22 15:40:36 +03:00
Lucy Wang 7efba10194 remove "sudo: false" now that travis no longer supports it 
https://changelog.travis-ci.com/deprecation-container-based-linux-build-environment-82037
 2019-03-22 15:40:36 +03:00
Vostretsov Nikita f56079f6c7 Test cleanups 
PEP8 fixes
no need to close implicitly
do not use pytest
need to put it into class
remove round-robin queue
additional check for empty queue
use pytest tmpdir fixture
 2019-03-22 15:40:36 +03:00
Adrián Chaves 9c314800e4 Document the SCRAPY_PROJECT environment variable 
Fixes #1109
 2019-03-22 15:40:36 +03:00
Adrián Chaves afdb69ea6d Add a troubleshooting section to the installation instructions 
Its initial content covers the workaround for #2473.
 2019-03-22 15:40:36 +03:00