Commit Graph

6808 Commits

Author SHA1 Message Date
Adrián Chaves 9c314800e4 Document the SCRAPY_PROJECT environment variable
Fixes #1109
2019-03-22 15:40:36 +03:00
Adrián Chaves afdb69ea6d Add a troubleshooting section to the installation instructions
Its initial content covers the workaround for #2473.
2019-03-22 15:40:36 +03:00
Vostretsov Nikita 821f5bb260 First implementation
handle exception
use O(N) instead of O(NlogN)
here we have request as struct
additional check for meptiness
small performance improvement
do not consume another request
test number of responses
mark requests
back to 3 slots test case
raise exceptions in case of missed meta
add marks to requests and work only with your own requests
only disk queue should obtain signals
separate functions for slot rasd/write
use signlas without variable
stop crawler
get signals in correct place
logic test for download-aware priority queue
update comment for structure
ensure text type
transform slot name to path
use implicit structure
use unicode type implicitly
use real crawler
add signals
more slot accounting
simple implementation of pop
small slot accounting code
no need for custom len function
ability to call super in py27
add slots
generic tests for downloader aware queue
dummy implementation of crawler aware priority queue
move common logic to base class
rename class
pass crawler to pqclass constructor
do not copy quelib.PriorityQueue code
add comment about new class
remove obsolete function
modify behaviour of queuelib.PriorityQueue to dodge very complex priority
better way to get name
remove obsolete commentary
check boundaries
function for priority convertion with known limits
correct import path
move file
do not switch on by deffault as ip concurrency not supported
set scheduler slot in case of empty slot
use constant
single place for added urls
single place for constants
use as default queue
correct format for error text
test migration from old version with on disk queue
in these tests we have only two inflection points - jobdir and priority_queue_cls
we do not need separate mock spider, use usual one
do not rely on order of dict elements, imply order of list
test round robiness of priority queue
add comments and requirements for our magick function
remove debug logging
put queues into slot
as we fabricate priorities we do not need special types anymore
fabricate priority for priority queue
more versatile priorities
Scheduler class is not inflection point
wrap correct types
check for emptinees before initialization
tests for new priority queue
correct default type for startprios
use exact values
put common settings to base class
test priorities for disk scheduler
test dequeue for disk scheduler
test length for disk scheduler
setUp/tearDown methods for on disk schedulers
new methods
remove excessive line
base class to handle scheduler creation
correct method names
test priorities
deque test
close scheduler on test end
enqueue some requests
test template for scheduler
use downloader slot
I/O implementation for RoundRobin queue
round-robin implementation without I/O and slot detection
wrappers for every disk queue class
2019-03-22 15:40:36 +03:00
Vostretsov Nikita a25cf5c82f function to get unique file queues for any type of base queue 2018-11-20 16:13:09 +00:00
Konstantin Lopuhin 886513c375
Merge pull request #3495 from toddrme2178/patch-1
[MRG+1] Include additional files in sdists
2018-11-20 11:10:32 +03:00
Todd 491929c212
Include additional files in sdists
In particular this includes files needed for running the tests, as well as the changelog.
2018-11-16 13:38:19 -05:00
Mikhail Korobov dc65e7527d
Merge pull request #3468 from ilhaoni/feature/update-docs-intro-tutorial
[MRG+1] Update Scrapy Tutorial docs
2018-11-06 03:33:09 +05:00
Immanuella Lim 6c98010f11 Remove 'Dive into Python3' reference 2018-11-04 16:04:45 +08:00
Immanuella Lim c9b5bd6ad7 Remove ad link Dive Into Python3 from tutorial docs 2018-10-18 02:22:32 +08:00
Daniel Graña 06f2db7fd1
Merge pull request #3445 from jfflisikowski/3097-item-urls-not-defined-in-example
[MRG+1] Updating debug example file (#3097)
2018-10-16 19:51:32 -03:00
jfflisikowski edaf74bfae Correct the unclear comments by adding <# < processing code not shown > 2018-10-02 19:48:48 +02:00
Daniel Graña bafd174a9f
Merge pull request #3390 from scrapy/parsel-1.5
[MRG+1] update Scrapy to use parsel 1.5
2018-09-18 13:07:41 -03:00
Mikhail Korobov ffbd33edac DOC mention gotcha with `foo::text` selector and empty `foo` elements
also, move "Selecting attributes" reference closer to `a::atr(href)` example
2018-09-18 05:03:35 +05:00
Mikhail Korobov 2c3b2158c9 DOC address @stummjr's review comments
* fixed several small issues
* re-written "Creating Selectors" section
* fixed remaining .extract usage in tests
2018-09-18 05:02:17 +05:00
Mikhail Korobov 9db21e5502 DOC fix remove_namespaces example
See https://github.com/scrapy/parsel/pull/119
2018-09-15 02:43:37 +05:00
Mikhail Korobov dc95ecbe25 DOC use autodocs for selectors; document more methods and attributes; suggest get/getall 2018-09-12 18:36:25 +05:00
Mikhail Korobov 7fdfdb7fa2 DOC reorganize selectors tutorial, port more topics from parsel docs, adjust wording in the introduction 2018-09-12 17:57:27 +05:00
Mikhail Korobov bdcc045f62 DOC switch from .extract to get/getall API in docs
Also, response.urljoin is added in a few places, for robustness.
2018-09-12 17:57:27 +05:00
Mikhail Korobov afce9716fa DOC mention .attrib in the tutorial 2018-09-12 17:57:27 +05:00
Mikhail Korobov 12e42bbe06 switch SgmlLinkExtractor to .getall 2018-09-12 17:57:27 +05:00
Mikhail Korobov 460f0f0451 [backwards incompatible] switch ItemLoader from .extract to .getall.
This change is backwards incompatible if ItemLoader is used with a custom Selector
subclass which overrides .extract without overriding .getall.
2018-09-12 17:57:27 +05:00
Mikhail Korobov 8c29be606c update spider templates to use .get 2018-09-12 17:57:27 +05:00
Mikhail Korobov 53da56c8dc TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
Mikhail Korobov 2c48d156db DOC cleanup references in tutorials:
* remove unused link
* fix ReST syntax
* fix a link to regular expression docs
2018-09-12 17:57:27 +05:00
Mikhail Korobov 09fd6c2a81 DOC unlink Firefox & Firebug sections from the tutorial for now.
See https://github.com/scrapy/scrapy/issues/3373 and https://github.com/scrapy/scrapy/issues/3372 for motivation.
2018-09-12 17:57:27 +05:00
Mikhail Korobov d32c4deaa9 DOC update Scrapy selectors tutorial to match parsel's tutorial better 2018-09-12 17:57:27 +05:00
Mikhail Korobov ca27010d4f DOC .extract_first() -> .get() 2018-09-12 17:57:27 +05:00
Mikhail Korobov 395d9d033a add pytest temp files to gitignore 2018-09-12 17:57:27 +05:00
Mikhail Korobov 0ccead6681 DOC more Python 3 in examples 2018-09-12 17:57:27 +05:00
Mikhail Korobov 25ac4691b4 require parsel 1.5+ 2018-09-12 17:57:27 +05:00
Daniel Graña 7223978594
Merge pull request #3381 from StasDeep/fix/issue-3380
[MRG+1] Use dont_filter=True for contracts requests
2018-09-06 11:41:02 -03:00
Daniel Graña 2aae514f99
Merge branch 'master' into fix/issue-3380 2018-09-05 12:25:07 -03:00
Daniel Graña 5176765589
Merge pull request #3377 from StasDeep/feature/issue-3364
Add handling of errors in contract methods
2018-09-05 12:24:50 -03:00
Daniel Graña d7b98a3714
Merge branch 'master' into feature/issue-3364 2018-09-05 11:26:59 -03:00
Daniel Graña ae8a0dc77c
Merge pull request #3383 from StasDeep/feature/issue-3382
[MRG+1] Add ability to use FormRequest in contracts
2018-09-05 11:21:26 -03:00
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot
[MRG+1] New signal for reqeuests reached downloader
2018-09-05 11:17:13 -03:00
Stas Glubokiy 8dbbbd1395 Use request_cls attribute in contract definition 2018-09-03 20:07:37 +03:00
Vostretsov Nikita c02cfa574c remove comma 2018-08-29 11:21:55 +00:00
Daniel Graña 0007cd032d
Merge pull request #3405 from stav/offsite-pep8
[MRG+1] PEP8 ofsite middleware
2018-08-28 14:51:39 -03:00
Steven Almeroth 6e9fa3a41f PEP8 ofsite middleware 2018-08-24 15:18:16 -04:00
Mikhail Korobov e45ef7dcd9
Merge pull request #3400 from testingcan/docs-developer-tools
[MRG+1] Added general guide for developer tools

Fixes #3373 and #3372.
2018-08-23 19:44:14 +05:00
Raphael Wuillemier 79de3d569a Removed obsolete firebug-images 2018-08-23 16:19:13 +02:00
testingcan e98e7f8506
Added missing curly brace 2018-08-23 14:50:49 +02:00
Raphael Wuillemier 4d3aaabbca Updated code, added code snippets and improved readability 2018-08-23 12:40:31 +02:00
testingcan 3a71e7dbce
Increased length of "=" 2018-08-22 16:57:51 +02:00
Raphael Wuillemier af555cab23 Added general guide for developer tools instead of Firefox and Firebug-sections 2018-08-22 14:15:53 +02:00
Stas Glubokiy 9ab85fe788 Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3382 2018-08-19 17:21:28 +03:00
Stas Glubokiy 57824600a8 Use six.get_unbound_function in test_same_url 2018-08-19 16:56:41 +03:00
Stas Glubokiy 0467737cf0 Fix mockserver usage 2018-08-18 15:43:46 +03:00
Stas Glubokiy 1311f6b536 Merge branch 'master' of github.com:StasDeep/scrapy into fix/issue-3380 2018-08-18 15:42:10 +03:00