RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,209 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

2639 Commits

Author SHA1 Message Date
Julia Medina 24a07fd892 Verify SPIDER_MANAGER_CLASS interface in CrawlerRunner 2015-04-10 13:43:33 -03:00
nyov 6d48c19acd dissolve `scrapy.contrib_exp` 2015-04-07 21:20:48 +00:00
Pablo Hoffman 4b11501eb5 Merge pull request #963 from tpeng/fix-xmliter-lxml 
[MRG+1] support namespace prefix in xmliter_lxml
 2015-04-07 18:13:01 -03:00
Daniel Graña 9706119336 Merge pull request #1128 from Curita/move-spider-settings-population-to-crawler 
[MRG+1] Move spider settings population from CrawlerRunner to Crawler.__init__
 2015-04-04 18:08:05 -03:00
Julia Medina 6f9265b44b Move spider settings population to Crawler.__init__ 2015-04-01 23:29:10 -03:00
nyov dc88be75e2 more python3 compatibility changes for urllib 2015-04-01 19:50:56 +00:00
Daniel Graña ec4251afd7 Merge pull request #1089 from drack3800/master 
[MRG+1] Add test for webclient with POST method and no body given
 2015-03-31 17:30:08 -03:00
Pablo Hoffman bb4c922d85 Merge pull request #1081 from scrapy/dict-items 
Allow spiders to return dicts.
 2015-03-27 15:19:27 -03:00
nyov 84edc2ebc7 Add Response.urljoin() testcase 
and add evaluation of base-url for HtmlResponse.
 2015-03-27 05:59:01 +00:00
drack3800 1b6d5a011a Added webclient test for checking Content-Length header in response for POST request with no given body 2015-03-21 04:02:51 +03:00
Mateusz Golewski f92bc09bf4 Add re_first() to SelectorList and iflatten() to utils.python 2015-03-18 21:11:08 -03:00
Mateusz Golewski 2742b4d8c2 Add tests to extract_first() 2015-03-18 21:11:08 -03:00
Nicolás Alejandro Ramírez Quiros ee82fe0e24 Merge pull request #1016 from SudShekhar/jsonProcessor 
[MRG+1] Added JmesSelect
 2015-03-18 08:11:25 -03:00
Mikhail Korobov 39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Pablo Hoffman 934584a355 Merge pull request #1020 from jojje/gzip_http_cache 
[MRG+1] add gzip compression to filesystem http cache backend
 2015-03-17 14:32:06 -03:00
Pablo Hoffman f924567591 Merge pull request #983 from ArturGaspar/linkextractor_css 
[MRG+1] CSS support in link extractors
 2015-03-17 01:07:47 -03:00
nramirezuy c13e23641b httpcache dont_cache meta #19 #689 2015-03-16 11:50:04 -03:00
Mikhail Korobov 645366a0fd Merge pull request #1070 from berkerpeksag/items-equal 
assertItemsEqual was renamed to assertCountEqual in Python 3.
 2015-03-08 22:54:57 +05:00
Berker Peksag c86e1bebec assertItemsEqual was renamed to assertCountEqual in Python 3. 2015-03-08 15:18:49 +02:00
Berker Peksag 31e5f164d4 Import unittest.mock if available. 
mock is in the stdlib since Python 3.3.
 2015-03-08 15:03:04 +02:00
Sudhanshu Shekhar 839ffba971 Added the first version of SelectJmes 
Utilizes jmespath. Also, added tests and documentation for the same.
 2015-02-24 22:59:01 +05:30
Jonas Tingeborn bd5d99a2d2 add gzip compression to filesystem http cache backend 2015-01-21 20:18:11 +01:00
Julia Medina d68615a5af Test the parse command locally instead of against an external url 2015-01-19 10:28:25 -03:00
tpeng 82d138e87e support namespace prefix in xmliter_lxml 2014-12-15 16:17:06 +01:00
Artur Gaspar 403fc686b8 tests for CSS support in link extractors 2014-12-11 18:20:30 -02:00
immerrr 82b187f283 S3DownloadHandler: fix auth for requests with quoted paths/query params 2014-12-11 18:14:36 +03:00
Lev Berman fdb6bb07c0 #977 - test dropping requests 2014-11-28 10:53:33 +03:00
Lev Berman e04b0aff74 An attempt to resolve #977, add signal to be sent when request is dropped by the scheduler 2014-11-27 15:10:15 +03:00
Pablo Hoffman c31fb87335 Merge pull request #954 from kalessin/int-download-timeout 
Force to read DOWNLOAD_TIMEOUT as int (for example to pass using environment variable)
 2014-11-26 17:14:18 -02:00
Pablo Hoffman dedea72774 Merge pull request #946 from tpeng/limit-response-size 
avoid download large response
 2014-11-25 17:57:55 -02:00
tpeng cd19382754 attemp to fix travis fails 2014-11-25 14:20:25 +01:00
Daniel Graña 8d8e1b2c0c mitmproxy 0.10.1 needs netlib 0.10.1 too 2014-11-21 12:15:02 -02:00
Daniel Graña 314db3db8b pin mitmproxy 0.10.1 as >0.11 does not work with tests 2014-11-21 10:54:43 -02:00
Martin Olveyra 7910fa0172 Force to read DOWNLOAD_TIMEOUT as int (for example to pass using 
environment variable)
 2014-11-21 01:09:32 -02:00
tpeng a69f042d10 add 2 more test cases and minor doc fixes 2014-11-19 15:31:07 +01:00
tpeng fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 2014-11-12 12:28:02 +01:00
Pablo Hoffman efe589c643 Merge pull request #882 from ahlen/feature/csvfeed-quotechar 
[MRG+1] Allow to specify the quotechar in CSVFeedSpider
 2014-11-04 11:32:59 -02:00
Mikhail Korobov 36eec8f413 dont_obey_robotstxt meta key; don't process requests to /robots.txt 2014-09-23 00:10:43 +06:00
Mikhail Korobov 49645d4bf9 TST small cleanup of a cookie test 2014-09-21 05:31:34 +06:00
Mikhail Korobov c543fe6e4c Merge pull request #878 from andrewshir/master 
Fix bug for ".local" host name
 2014-09-21 05:24:54 +06:00
andrewshir e583c030db Test for local domains (without dots) added 2014-09-14 14:24:16 +06:00
Mikael Åhlén 22da1783bd added a test-case for wrong quotechar 2014-09-13 03:47:40 +02:00
Mikael Åhlén 47b6dff9f1 Allow to specify the quotechar in CSVFeedSpider 2014-09-13 02:14:57 +02:00
Daniel Graña 5bcabfe9c9 SPIDER_MODULES can be set as a csv string 2014-09-10 23:25:57 -03:00
Daniel Graña ec93c0fdcc Add the tests changes for previous commit 2014-09-10 12:05:18 -03:00
Daniel Graña 99971dc8a8 Do not pop the crawler from the managed list 2014-09-09 20:59:07 +00:00
Julia Medina c2592b39fd Test verifying that CrawlerRunner populates spider class settings 2014-09-01 21:56:57 -03:00
Julia Medina 9ef3972cfb Per-spider settings tests 2014-09-01 21:56:57 -03:00
Daniel Graña ccde3317d7 Merge pull request #816 from Curita/api-cleanup 
GSoC API cleanup
 2014-09-01 21:55:36 -03:00
yakxxx e4689556f0 SgmlLinkExtractor - fix for parsing <area> tag with Unicode present 2014-08-28 18:55:58 +02:00
nramirezuy 2a540206a7 fix xmliter namespace on selected node 2014-08-19 15:44:49 -03:00
Daniel Graña d684ecad7b Merge pull request #846 from rocioar/master 
fix dont_merge_cookies bad behaviour when set to false on meta
 2014-08-18 13:54:11 -03:00
Daniel Graña a9292cfab7 jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository 2014-08-15 23:28:13 -03:00
Rocio Aramberri 51b0bd281d fix dont settings on meta behaviour, add docs and tests 2014-08-15 13:47:42 -07:00
Julia Medina 70f2010db1 Change error type when updating frozen settings 2014-08-14 11:59:25 -03:00
Julia Medina 419026615f Deprecate Crawler.spiders attribute 2014-08-14 09:19:41 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Julia Medina 900a487682 Support multiple simultaneous LogObservers listening different crawlers 2014-08-12 14:02:56 -03:00
Julia Medina 870438e5f4 Update tests utils, fixing get_crawler and removing docrawl 2014-08-12 14:02:56 -03:00
Julia Medina d7038b2a13 SpiderManager interface cleanup 2014-08-12 14:02:55 -03:00
Julia Medina 3ae971468f Add Settings.copy, freeze and frozencopy method 2014-08-12 14:02:55 -03:00
Julia Medina a995727117 Connect spider_closed signal after a crawler is bound to a Spider 2014-08-12 14:02:55 -03:00
Julia Medina eb0253e530 Update from_crawler method as well as set_crawler on CrawlSpider 2014-08-11 11:24:01 -03:00
Julia Medina 84fa004793 Add from_crawler class method to base Spider 2014-08-11 11:23:57 -03:00
Pablo Hoffman 69a665eb32 Merge pull request #832 from ivannotes/master 
Bugfix for leaking Proxy-Authorization header to targeted host
 2014-08-04 15:20:51 -03:00
Daniel Graña 511a26978f Merge pull request #817 from nramirezuy/fix-startproject-project-name 
[MRG] Fix startproject project name
 2014-08-04 11:55:35 -03:00
Mikhail Korobov e62bbf0766 PY3 top-level shortcuts work 2014-08-02 00:24:32 +06:00
Daniel Graña ee21eaa6f3 Merge pull request #835 from scrapy/modernize-asserts 
[MRG] modernize some of the asserts
 2014-08-01 15:19:36 -03:00
Mikhail Korobov 1f59a69a97 PY3 port scrapy.link 2014-08-02 00:16:01 +06:00
Mikhail Korobov 628a8d8a15 TST modernize some of the asserts 2014-08-01 23:44:23 +06:00
Nuno Maximiano 08224c92f4 add project name validation 2014-08-01 14:36:30 -03:00
Mikhail Korobov e2c1226838 PY3 port scrapy.contrib.feedexport 2014-08-01 23:28:28 +06:00
Mikhail Korobov 51bd9f7d64 TST enable doctests for Python 3 2014-08-01 23:12:52 +06:00
Mikhail Korobov 444052a2f9 PY3 port scrapy.utils.misc.arg_to_iter; fix scrapy.utils.misc.md5sum doctest 2014-08-01 22:56:03 +06:00
Mikhail Korobov 83432184c8 PY3 enable more tests 2014-08-01 16:53:18 +06:00
Mikhail Korobov 669b62c741 PY3 port scrapy.utils.spider 2014-08-01 16:43:39 +06:00
Mikhail Korobov 560c32d330 PY3 fix test_utils_sitemap. 
Sitemap class is only initialized with bytes by Scrapy;
broken tests was only a testing issue.
 2014-08-01 16:17:40 +06:00
Mikhail Korobov 330743411e PY3 enable more tests 2014-08-01 16:10:34 +06:00
Mikhail Korobov 7183a87c35 Merge pull request #828 from dangra/string-headers 
normalize header values to bytes (and PY3)
 2014-08-01 15:51:57 +06:00
Daniel Graña ccae49813a PY3: port scrapy/utils/httpobj.py 2014-08-01 02:38:29 -03:00
Daniel Graña c564334d00 PY3: port scrapy/utils/gz.py 2014-08-01 02:25:33 -03:00
Daniel Graña 910d0d36c1 PY3: port scrapy/utils/conf.py 2014-08-01 02:20:34 -03:00
Daniel Graña 521303e27f fix ftp test cases 2014-08-01 02:10:57 -03:00
Daniel Graña f42b44b50e normalize header values to bytes 2014-08-01 00:02:50 -03:00
ivannotes 484a0159d0 Add test case for tunneling proxy 2014-08-01 09:43:17 +08:00
Daniel Graña 7cd271eb8e remove more references to runtests.sh 2014-07-31 17:30:45 -03:00
Daniel Graña 70ee66ba0c Collect the list of packages not working on python3 2014-07-31 14:06:34 -03:00
Daniel Graña caab9f29f2 move tests requirements under tests dir 2014-07-30 19:45:08 -03:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00