RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,808 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

812 Commits

Author SHA1 Message Date
Vostretsov Nikita 821f5bb260 First implementation 
handle exception
use O(N) instead of O(NlogN)
here we have request as struct
additional check for meptiness
small performance improvement
do not consume another request
test number of responses
mark requests
back to 3 slots test case
raise exceptions in case of missed meta
add marks to requests and work only with your own requests
only disk queue should obtain signals
separate functions for slot rasd/write
use signlas without variable
stop crawler
get signals in correct place
logic test for download-aware priority queue
update comment for structure
ensure text type
transform slot name to path
use implicit structure
use unicode type implicitly
use real crawler
add signals
more slot accounting
simple implementation of pop
small slot accounting code
no need for custom len function
ability to call super in py27
add slots
generic tests for downloader aware queue
dummy implementation of crawler aware priority queue
move common logic to base class
rename class
pass crawler to pqclass constructor
do not copy quelib.PriorityQueue code
add comment about new class
remove obsolete function
modify behaviour of queuelib.PriorityQueue to dodge very complex priority
better way to get name
remove obsolete commentary
check boundaries
function for priority convertion with known limits
correct import path
move file
do not switch on by deffault as ip concurrency not supported
set scheduler slot in case of empty slot
use constant
single place for added urls
single place for constants
use as default queue
correct format for error text
test migration from old version with on disk queue
in these tests we have only two inflection points - jobdir and priority_queue_cls
we do not need separate mock spider, use usual one
do not rely on order of dict elements, imply order of list
test round robiness of priority queue
add comments and requirements for our magick function
remove debug logging
put queues into slot
as we fabricate priorities we do not need special types anymore
fabricate priority for priority queue
more versatile priorities
Scheduler class is not inflection point
wrap correct types
check for emptinees before initialization
tests for new priority queue
correct default type for startprios
use exact values
put common settings to base class
test priorities for disk scheduler
test dequeue for disk scheduler
test length for disk scheduler
setUp/tearDown methods for on disk schedulers
new methods
remove excessive line
base class to handle scheduler creation
correct method names
test priorities
deque test
close scheduler on test end
enqueue some requests
test template for scheduler
use downloader slot
I/O implementation for RoundRobin queue
round-robin implementation without I/O and slot detection
wrappers for every disk queue class
 2019-03-22 15:40:36 +03:00
Mikhail Korobov 2c3b2158c9 DOC address @stummjr's review comments 
* fixed several small issues
* re-written "Creating Selectors" section
* fixed remaining .extract usage in tests
 2018-09-18 05:02:17 +05:00
Mikhail Korobov 53da56c8dc TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
Daniel Graña 7223978594
Merge pull request #3381 from StasDeep/fix/issue-3380 
[MRG+1] Use dont_filter=True for contracts requests
 2018-09-06 11:41:02 -03:00
Daniel Graña 2aae514f99
Merge branch 'master' into fix/issue-3380 2018-09-05 12:25:07 -03:00
Daniel Graña d7b98a3714
Merge branch 'master' into feature/issue-3364 2018-09-05 11:26:59 -03:00
Daniel Graña ae8a0dc77c
Merge pull request #3383 from StasDeep/feature/issue-3382 
[MRG+1] Add ability to use FormRequest in contracts
 2018-09-05 11:21:26 -03:00
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot 
[MRG+1] New signal for reqeuests reached downloader
 2018-09-05 11:17:13 -03:00
Stas Glubokiy 8dbbbd1395 Use request_cls attribute in contract definition 2018-09-03 20:07:37 +03:00
Stas Glubokiy 9ab85fe788 Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3382 2018-08-19 17:21:28 +03:00
Stas Glubokiy 57824600a8 Use six.get_unbound_function in test_same_url 2018-08-19 16:56:41 +03:00
Stas Glubokiy 0467737cf0 Fix mockserver usage 2018-08-18 15:43:46 +03:00
Stas Glubokiy 1311f6b536 Merge branch 'master' of github.com:StasDeep/scrapy into fix/issue-3380 2018-08-18 15:42:10 +03:00
Stas Glubokiy 11576f5c8f Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3364 2018-08-18 15:24:07 +03:00
Vostretsov Nikita afb1458bd3 tests for new signal 2018-08-17 14:39:54 +00:00
Mikhail Korobov c7654f7cb1
Merge pull request #3379 from StasDeep/fix/issue-3378 
Use inspect.getmembers in tested_methods_from_spidercls
 2018-08-17 19:25:41 +05:00
Stas Glubokiy ddd69f4c10 Use MockServer in test_same_url 2018-08-15 20:39:43 +03:00
Stas Glubokiy 2cb4decb6a Move TestSameUrlSpider to test method 2018-08-15 20:36:10 +03:00
Stas Glubokiy 4de493efdd Add test_same_url 2018-08-15 20:24:00 +03:00
Daniel Graña 38608bc249 Use ignore_errors option from rmtree 2018-08-15 11:59:09 -03:00
Daniel Graña 4eaf8690b1 Twisted's unittest.Testcase assertRaiess can't be used as context manager 2018-08-15 08:54:18 -03:00
Daniel Grana a304d6b692 Workaround to pass tests/test_feedexporter.py under windows 2018-08-15 02:23:07 -07:00
Daniel Grana e7fe243c3e Fix test_crawler under windows 2018-08-15 01:09:23 -07:00
Daniel Grana 96517cb7de Fix test_command_parse under windows 2018-08-15 01:08:40 -07:00
Daniel Graña d93d960319 Fix test_utils_project under Windows 2018-08-15 01:53:20 -03:00
Jakob de Maeyer cb28175750 Fix csviter tests by explicitly using newline only 2018-08-15 01:35:01 -03:00
Daniel Graña ca53a8699a Fix presentation of template directory in startproject command 2018-08-15 01:35:01 -03:00
Daniel Graña 0e532e3dd8 Creating a connection to 0.0.0.0 fails on windows but not on linux nor mac 2018-08-15 01:35:01 -03:00
Daniel Graña a21abac743 fix ftp tests on windows 2018-08-15 01:35:01 -03:00
Daniel Graña fb09148c91 Fix bad merge on ParseCommandTest 2018-08-15 01:35:01 -03:00
Jakob de Maeyer 22505a34a9 Fix cmdline profiling test on Windows by using proper path composing 2018-08-15 01:35:01 -03:00
Jakob de Maeyer 034152961d Fix Feedexport test in Windows by using proper file URI 2018-08-15 01:35:01 -03:00
Daniel Graña ed8255bde0 Fix merge issues with stderr/out fixes for windows buffering 2018-08-15 01:33:38 -03:00
Jakob de Maeyer 57a1d66c61 Fix test issues caused by Windows pipe buffer filling up 2018-08-15 01:33:38 -03:00
Jakob de Maeyer 152fde70b1 Fix FTPTestCase by using Windows-friendly temporary file name 2018-08-15 01:33:38 -03:00
Daniel Graña 19ad94105f pywin32 is required to run tests under windows 2018-08-15 01:33:38 -03:00
Daniel Graña 4c53957f5b Skip leveldb tests on windows 2018-08-15 01:33:38 -03:00
Stas Glubokiy b4b1e48343 Add ability to use FormRequest in contracts 2018-08-11 22:18:43 +03:00
Stas Glubokiy 76220e8733 Use inspect.getmembers in tested_methods_from_spidercls 2018-08-11 18:49:12 +03:00
Stas Glubokiy 8bc536d88b Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3364 2018-08-11 18:34:37 +03:00
Stas Glubokiy ebbde57eca Add custom contracts tests 2018-08-11 18:21:07 +03:00
Stas Glubokiy 16dad81715 Fix contract errback 2018-08-09 21:07:25 +03:00
Daniel Graña db714f5a07
Merge pull request #3283 from CCInCharge/issue3247 
Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys
 2018-08-02 14:49:14 -03:00
Eugenio Lacuesta 9e14f8c7e4 Fix test for dupefilter 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 0089a4ab31 Add test for direct creation of dupefilter (no from_crawler/from_settings) 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta d306fe30ac Test dupefilter creation by the Scheduler 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 701cd2ff9d Add from_crawler support to dupefilters 2018-07-26 16:24:58 -03:00
CCInCharge 782f866572 Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys 2018-07-25 21:46:42 -07:00
Daniel Graña b6abd45926
Merge pull request #3342 from elacuesta/copy-request-flags 
[MRG+1] Include flags when copying requests
 2018-07-25 11:58:25 -03:00
Eugenio Lacuesta 917c1fde68 Merge branch 'master' into enhancement/alternate-feedexport-constructors 2018-07-20 12:09:50 -03:00