RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 153 MiB
Commit Graph

9291 Commits

Author SHA1 Message Date
Vostretsov Nikita 821f5bb260 First implementation 
handle exception
use O(N) instead of O(NlogN)
here we have request as struct
additional check for meptiness
small performance improvement
do not consume another request
test number of responses
mark requests
back to 3 slots test case
raise exceptions in case of missed meta
add marks to requests and work only with your own requests
only disk queue should obtain signals
separate functions for slot rasd/write
use signlas without variable
stop crawler
get signals in correct place
logic test for download-aware priority queue
update comment for structure
ensure text type
transform slot name to path
use implicit structure
use unicode type implicitly
use real crawler
add signals
more slot accounting
simple implementation of pop
small slot accounting code
no need for custom len function
ability to call super in py27
add slots
generic tests for downloader aware queue
dummy implementation of crawler aware priority queue
move common logic to base class
rename class
pass crawler to pqclass constructor
do not copy quelib.PriorityQueue code
add comment about new class
remove obsolete function
modify behaviour of queuelib.PriorityQueue to dodge very complex priority
better way to get name
remove obsolete commentary
check boundaries
function for priority convertion with known limits
correct import path
move file
do not switch on by deffault as ip concurrency not supported
set scheduler slot in case of empty slot
use constant
single place for added urls
single place for constants
use as default queue
correct format for error text
test migration from old version with on disk queue
in these tests we have only two inflection points - jobdir and priority_queue_cls
we do not need separate mock spider, use usual one
do not rely on order of dict elements, imply order of list
test round robiness of priority queue
add comments and requirements for our magick function
remove debug logging
put queues into slot
as we fabricate priorities we do not need special types anymore
fabricate priority for priority queue
more versatile priorities
Scheduler class is not inflection point
wrap correct types
check for emptinees before initialization
tests for new priority queue
correct default type for startprios
use exact values
put common settings to base class
test priorities for disk scheduler
test dequeue for disk scheduler
test length for disk scheduler
setUp/tearDown methods for on disk schedulers
new methods
remove excessive line
base class to handle scheduler creation
correct method names
test priorities
deque test
close scheduler on test end
enqueue some requests
test template for scheduler
use downloader slot
I/O implementation for RoundRobin queue
round-robin implementation without I/O and slot detection
wrappers for every disk queue class
 2019-03-22 15:40:36 +03:00
Marc Hernández 70aa5b1333
Fix numeration 2019-03-20 15:32:20 +01:00
Maram Sumanth 4c89e53e68
Update test_http_request.py 2019-03-20 18:46:25 +05:30
Maram Sumanth 282f24c510
Update form.py 2019-03-20 18:46:22 +05:30
Anubhav Patel 044318920a doc for creating custom cache storage backend. 2019-03-17 16:54:28 +05:30
Eugenio Lacuesta 92bbc5290d Rule.process_request - Renaming 2019-03-16 13:19:38 +00:00
Eugenio Lacuesta 6760bca74b Rename Request.kwargs to Request.cb_kwargs 2019-03-15 22:42:29 +00:00
Eugenio Lacuesta 645e8d16a4 Count keyword argument checks 2019-03-15 22:20:36 +00:00
Eugenio Lacuesta 01ed605d02 PEP8 changes to test_spider.py 2019-03-15 16:54:14 +00:00
Victor Torres fda1d04b81
Merge branch 'master' into feed-storage-s3-acl 2019-03-15 13:26:45 -03:00
Adrián Chaves d346b8cb0f
Merge pull request #3626 from Gallaecio/middlewares-from-crawler 
[MRG+1] Document that the main entry point of downloader and spider middlewar…
 2019-03-15 08:58:35 +01:00
Adrián Chaves 70a0f1f2e4
Merge pull request #3610 from Gallaecio/spidercls-check 
[MRG+1] Check that spidercls arguments in scrapy.crawler classes are not spid…
 2019-03-15 08:25:16 +01:00
Adrián Chaves 1f7413dc88
Merge pull request #3635 from matthieucham/feature-filteringlinkextractor-restrict-text 
[MRG+1] Feature filteringlinkextractor restrict text
 2019-03-15 07:58:28 +01:00
Adrián Chaves 2bd5790d9b
Merge pull request #3629 from johndela1/master 
[MRG+1] rel_has_nofollow: remove redundant if statement
 2019-03-15 07:53:23 +01:00
Adrián Chaves a2ff647aac Remove docs/topics/ubuntu.rst 2019-03-15 06:36:15 +01:00
Anubhav Patel 66a502db8c
Merge branch 'master' into logFormatter-doc-patch 2019-03-15 00:28:54 +05:30
Mikhail Korobov 09e56ae43e
Merge pull request #3648 from Gallaecio/backward 
backwards → backward (adj.)
 2019-03-14 22:21:34 +05:00
Mikhail Korobov 5dc94db847
Merge branch 'master' into backward 2019-03-14 22:21:09 +05:00
Mikhail Korobov 54d106033f
Merge pull request #3649 from Gallaecio/inline-code-formatting 
[MRG+1] Switch from ` to `` where inline code formatting is desired
 2019-03-14 22:18:57 +05:00
Mikhail Korobov 7c54479a0e
Merge pull request #3662 from anubhavp28/link-patch 
Fix a link inside docs
 2019-03-14 22:18:04 +05:00
Mikhail Korobov c3b6755671
Merge pull request #3668 from Gallaecio/retry-complete-docs 
Remove the unexisting retry_complete signal from the documentation
 2019-03-14 22:16:28 +05:00
Mikhail Korobov f439553844
Merge pull request #3670 from Gallaecio/depth-first-doc 
Clarify the documentation of DEPTH_PRIORITY further
 2019-03-14 22:10:42 +05:00
Mikhail Korobov 9c2c312451
Merge pull request #3673 from Gallaecio/friendlier-documentation-1 
Use the description from README.rst on index.rst
 2019-03-14 21:53:44 +05:00
Mikhail Korobov 39a17aff3a
Merge pull request #3676 from Gallaecio/patch-1 
Update developer-tools.rst
 2019-03-14 21:52:37 +05:00
Eugenio Lacuesta 83ec947fe7 Rule.process_request defaults to None in the docs 2019-03-13 11:23:51 +00:00
Eugenio Lacuesta b30ca379b6 Rule.process_request: docs 2019-03-13 11:02:51 +00:00
Eugenio Lacuesta 22fda61d62 Rule.process_request: tests 2019-03-13 10:54:38 +00:00
Eugenio Lacuesta 43fd622968 Rule.process_request: optionally take a Response object 2019-03-13 10:43:44 +00:00
Maram Sumanth 35f7595dbe
changed variable names 2019-03-11 23:58:37 +05:30
Anubhav Patel e9cd4ee03a fix list alignment and line width 2019-03-10 20:37:56 +05:30
Anubhav Patel 82049e9c41 make suggested changes. 2019-03-10 20:17:07 +05:30
Adrián Chaves 91aec8b3bb
Update developer-tools.rst 
Fixes #3674
 2019-03-08 18:19:30 +01:00
Adrián Chaves b1063d9b2c Use the description from README.rst on index.rst 2019-03-08 17:22:49 +01:00
Adrián Chaves e108e3adbf Clarify the documentation of DEPTH_PRIORITY further 2019-03-08 15:13:11 +01:00
Adrián Chaves 4ef38d925e Remove the unexisting retry_complete signal from the documentation 2019-03-08 14:21:00 +01:00
Adrián Chaves 120007c057 Add a FAQ entry on how to deal with long lists of allowed domains 2019-03-08 13:53:47 +01:00
Anubhav Patel 924b67437b move api docs to source code 2019-03-07 16:40:59 +05:30
Anubhav Patel 184def1060 fix a link inside docs 2019-03-07 00:09:10 +05:30
Maram Sumanth f7bf3abbd0
Modified code 2019-03-06 14:10:03 +05:30
Anubhav Patel 82d239f3b1 docs for scrapy.logformatter 2019-03-06 12:08:09 +05:30
Maram Sumanth 7da460b793
Update form.py 2019-03-04 17:25:15 +05:30
Maram Sumanth 8831fafabc
Update test_http_request.py 2019-03-04 15:42:48 +05:30
Maram Sumanth fdf03a6d0d
correcting tests 2019-03-04 15:12:44 +05:30
Maram Sumanth d75b61b96a
Update test_http_request.py 2019-03-04 15:07:12 +05:30
Maram Sumanth 6eca6f92c6
Update form.py 2019-03-04 14:59:34 +05:30
Adrián Chaves 75d6f56c8a Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
Adrián Chaves 858f5be747 backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
Matthieu Grandrie e3b15252c8 New constructor arg *restrict_text* for FilteringLinkExtractor. 
Same as allow and deny args, it holds a string, a regex or an iterable of. Links whose text don't match one of the regex are filtered out.
DOC restrict_text in LxmlLinkExtractor
 2019-02-28 17:21:17 +01:00
Júlio César Batista 0bb3d8ca93 Updating Google Cloud Storage scheme to gs instead of gcs 2019-02-27 18:41:01 -03:00
Daniel Graña c72ab1d4ba
Merge pull request #3625 from Ordepsousa/master 
[MRG+1] A different S3 Endpoint URL is now possible when uploading images
 2019-02-20 11:39:21 -03:00