Commit Graph

6819 Commits

Author SHA1 Message Date
Vostretsov Nikita 4163a7a1c7 no need for this 2019-03-22 15:40:37 +03:00
Daniel Graña 6ff2574c27 Needs to be installed within tox env 2019-03-22 15:40:36 +03:00
Daniel Graña 7d3175ac84 Fix boto import error under Jessie testing environment 2019-03-22 15:40:36 +03:00
Vostretsov Nikita d970be64cc Integration test
integration testing only everything is working, not logic of PQ
use method
create slot attribute in constructor
corect class for test case
stop crawler in teardown method
use class
correct entity naming
python 2 adaptation
integration test with crawler and spider
2019-03-22 15:40:36 +03:00
Vostretsov Nikita a23e1894b3 Fix boto problem
another way to fix boto problem
Revert "fix for travis ci based on https://github.com/boto/boto/issues/3717"

This reverts commit 150d2564ff0ea994652da7f5be333d72e0b38d93.
fix for travis ci based on https://github.com/boto/boto/issues/3717
2019-03-22 15:40:36 +03:00
Vostretsov Nikita a46613afa8 use regular comments 2019-03-22 15:40:36 +03:00
Vostretsov Nikita 6af964cc0b common indentation for comment 2019-03-22 15:40:36 +03:00
Vostretsov Nikita 484927b08c less complex implementation 2019-03-22 15:40:36 +03:00
Vostretsov Nikita 0e06b9a816 use urlparse_cached where it is possible 2019-03-22 15:40:36 +03:00
Lucy Wang 7efba10194 remove "sudo: false" now that travis no longer supports it
https://changelog.travis-ci.com/deprecation-container-based-linux-build-environment-82037
2019-03-22 15:40:36 +03:00
Vostretsov Nikita f56079f6c7 Test cleanups
PEP8 fixes
no need to close implicitly
do not use pytest
need to put it into class
remove round-robin queue
additional check for empty queue
use pytest tmpdir fixture
2019-03-22 15:40:36 +03:00
Adrián Chaves 9c314800e4 Document the SCRAPY_PROJECT environment variable
Fixes #1109
2019-03-22 15:40:36 +03:00
Adrián Chaves afdb69ea6d Add a troubleshooting section to the installation instructions
Its initial content covers the workaround for #2473.
2019-03-22 15:40:36 +03:00
Vostretsov Nikita 821f5bb260 First implementation
handle exception
use O(N) instead of O(NlogN)
here we have request as struct
additional check for meptiness
small performance improvement
do not consume another request
test number of responses
mark requests
back to 3 slots test case
raise exceptions in case of missed meta
add marks to requests and work only with your own requests
only disk queue should obtain signals
separate functions for slot rasd/write
use signlas without variable
stop crawler
get signals in correct place
logic test for download-aware priority queue
update comment for structure
ensure text type
transform slot name to path
use implicit structure
use unicode type implicitly
use real crawler
add signals
more slot accounting
simple implementation of pop
small slot accounting code
no need for custom len function
ability to call super in py27
add slots
generic tests for downloader aware queue
dummy implementation of crawler aware priority queue
move common logic to base class
rename class
pass crawler to pqclass constructor
do not copy quelib.PriorityQueue code
add comment about new class
remove obsolete function
modify behaviour of queuelib.PriorityQueue to dodge very complex priority
better way to get name
remove obsolete commentary
check boundaries
function for priority convertion with known limits
correct import path
move file
do not switch on by deffault as ip concurrency not supported
set scheduler slot in case of empty slot
use constant
single place for added urls
single place for constants
use as default queue
correct format for error text
test migration from old version with on disk queue
in these tests we have only two inflection points - jobdir and priority_queue_cls
we do not need separate mock spider, use usual one
do not rely on order of dict elements, imply order of list
test round robiness of priority queue
add comments and requirements for our magick function
remove debug logging
put queues into slot
as we fabricate priorities we do not need special types anymore
fabricate priority for priority queue
more versatile priorities
Scheduler class is not inflection point
wrap correct types
check for emptinees before initialization
tests for new priority queue
correct default type for startprios
use exact values
put common settings to base class
test priorities for disk scheduler
test dequeue for disk scheduler
test length for disk scheduler
setUp/tearDown methods for on disk schedulers
new methods
remove excessive line
base class to handle scheduler creation
correct method names
test priorities
deque test
close scheduler on test end
enqueue some requests
test template for scheduler
use downloader slot
I/O implementation for RoundRobin queue
round-robin implementation without I/O and slot detection
wrappers for every disk queue class
2019-03-22 15:40:36 +03:00
Vostretsov Nikita a25cf5c82f function to get unique file queues for any type of base queue 2018-11-20 16:13:09 +00:00
Konstantin Lopuhin 886513c375
Merge pull request #3495 from toddrme2178/patch-1
[MRG+1] Include additional files in sdists
2018-11-20 11:10:32 +03:00
Todd 491929c212
Include additional files in sdists
In particular this includes files needed for running the tests, as well as the changelog.
2018-11-16 13:38:19 -05:00
Mikhail Korobov dc65e7527d
Merge pull request #3468 from ilhaoni/feature/update-docs-intro-tutorial
[MRG+1] Update Scrapy Tutorial docs
2018-11-06 03:33:09 +05:00
Immanuella Lim 6c98010f11 Remove 'Dive into Python3' reference 2018-11-04 16:04:45 +08:00
Immanuella Lim c9b5bd6ad7 Remove ad link Dive Into Python3 from tutorial docs 2018-10-18 02:22:32 +08:00
Daniel Graña 06f2db7fd1
Merge pull request #3445 from jfflisikowski/3097-item-urls-not-defined-in-example
[MRG+1] Updating debug example file (#3097)
2018-10-16 19:51:32 -03:00
jfflisikowski edaf74bfae Correct the unclear comments by adding <# < processing code not shown > 2018-10-02 19:48:48 +02:00
Daniel Graña bafd174a9f
Merge pull request #3390 from scrapy/parsel-1.5
[MRG+1] update Scrapy to use parsel 1.5
2018-09-18 13:07:41 -03:00
Mikhail Korobov ffbd33edac DOC mention gotcha with `foo::text` selector and empty `foo` elements
also, move "Selecting attributes" reference closer to `a::atr(href)` example
2018-09-18 05:03:35 +05:00
Mikhail Korobov 2c3b2158c9 DOC address @stummjr's review comments
* fixed several small issues
* re-written "Creating Selectors" section
* fixed remaining .extract usage in tests
2018-09-18 05:02:17 +05:00
Mikhail Korobov 9db21e5502 DOC fix remove_namespaces example
See https://github.com/scrapy/parsel/pull/119
2018-09-15 02:43:37 +05:00
Mikhail Korobov dc95ecbe25 DOC use autodocs for selectors; document more methods and attributes; suggest get/getall 2018-09-12 18:36:25 +05:00
Mikhail Korobov 7fdfdb7fa2 DOC reorganize selectors tutorial, port more topics from parsel docs, adjust wording in the introduction 2018-09-12 17:57:27 +05:00
Mikhail Korobov bdcc045f62 DOC switch from .extract to get/getall API in docs
Also, response.urljoin is added in a few places, for robustness.
2018-09-12 17:57:27 +05:00
Mikhail Korobov afce9716fa DOC mention .attrib in the tutorial 2018-09-12 17:57:27 +05:00
Mikhail Korobov 12e42bbe06 switch SgmlLinkExtractor to .getall 2018-09-12 17:57:27 +05:00
Mikhail Korobov 460f0f0451 [backwards incompatible] switch ItemLoader from .extract to .getall.
This change is backwards incompatible if ItemLoader is used with a custom Selector
subclass which overrides .extract without overriding .getall.
2018-09-12 17:57:27 +05:00
Mikhail Korobov 8c29be606c update spider templates to use .get 2018-09-12 17:57:27 +05:00
Mikhail Korobov 53da56c8dc TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
Mikhail Korobov 2c48d156db DOC cleanup references in tutorials:
* remove unused link
* fix ReST syntax
* fix a link to regular expression docs
2018-09-12 17:57:27 +05:00
Mikhail Korobov 09fd6c2a81 DOC unlink Firefox & Firebug sections from the tutorial for now.
See https://github.com/scrapy/scrapy/issues/3373 and https://github.com/scrapy/scrapy/issues/3372 for motivation.
2018-09-12 17:57:27 +05:00
Mikhail Korobov d32c4deaa9 DOC update Scrapy selectors tutorial to match parsel's tutorial better 2018-09-12 17:57:27 +05:00
Mikhail Korobov ca27010d4f DOC .extract_first() -> .get() 2018-09-12 17:57:27 +05:00
Mikhail Korobov 395d9d033a add pytest temp files to gitignore 2018-09-12 17:57:27 +05:00
Mikhail Korobov 0ccead6681 DOC more Python 3 in examples 2018-09-12 17:57:27 +05:00
Mikhail Korobov 25ac4691b4 require parsel 1.5+ 2018-09-12 17:57:27 +05:00
Daniel Graña 7223978594
Merge pull request #3381 from StasDeep/fix/issue-3380
[MRG+1] Use dont_filter=True for contracts requests
2018-09-06 11:41:02 -03:00
Daniel Graña 2aae514f99
Merge branch 'master' into fix/issue-3380 2018-09-05 12:25:07 -03:00
Daniel Graña 5176765589
Merge pull request #3377 from StasDeep/feature/issue-3364
Add handling of errors in contract methods
2018-09-05 12:24:50 -03:00
Daniel Graña d7b98a3714
Merge branch 'master' into feature/issue-3364 2018-09-05 11:26:59 -03:00
Daniel Graña ae8a0dc77c
Merge pull request #3383 from StasDeep/feature/issue-3382
[MRG+1] Add ability to use FormRequest in contracts
2018-09-05 11:21:26 -03:00
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot
[MRG+1] New signal for reqeuests reached downloader
2018-09-05 11:17:13 -03:00
Stas Glubokiy 8dbbbd1395 Use request_cls attribute in contract definition 2018-09-03 20:07:37 +03:00
Vostretsov Nikita c02cfa574c remove comma 2018-08-29 11:21:55 +00:00
Daniel Graña 0007cd032d
Merge pull request #3405 from stav/offsite-pep8
[MRG+1] PEP8 ofsite middleware
2018-08-28 14:51:39 -03:00