RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,155 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

5155 Commits

Author SHA1 Message Date
Jakob de Maeyer 4e40377bcb Allow multiple calls to Crawler.crawl() 2015-11-03 23:32:02 +01:00
Jakob de Maeyer 86c74ce53e Allow Spider.update_settings() to be an instance method 2015-11-02 16:57:57 +01:00
Jakob de Maeyer b06a670777 Initialize spider before calling its update_settings() 2015-11-02 16:47:07 +01:00
Jakob de Maeyer d67f292d92 Move Spider.update_settings() into Crawler.crawl() 2015-11-02 16:36:33 +01:00
Jakob de Maeyer c0566b2b07 Move extension init into Crawler.crawl() 2015-11-02 16:18:56 +01:00
Julia Medina 57f87b95d4 Merge pull request #1528 from scrapy/create-crawler 
public Crawler.create_crawler method
 2015-10-30 12:36:35 -03:00
Julia Medina 9424ca0fdb Merge pull request #1570 from jdemaeyer/tests/complete-settings-tests 
[MRG+1] Add some missing tests for scrapy.settings
 2015-10-30 12:36:25 -03:00
Mikhail Korobov 0000b6e9ef TST cleanup scrapy.utils.test.get_crawler 2015-10-30 20:27:03 +05:00
Mikhail Korobov a49c82ad62 TST improve CrawlerRunner tests 
* use CrawlerRunner.create_crawler instead of get_crawler helper in test_crawl;
* add a test for loading spiders by name;
* add a test for passing Crawler objects instead of Spider objects;
* add a test for CrawlerRunner.join
 2015-10-30 20:24:37 +05:00
Mikhail Korobov 11b11c9803 CrawlerRunner.create_crawler method 2015-10-30 20:22:31 +05:00
Julia Medina 72eeead6db Merge pull request #1524 from Digenis/relocate_telnet_console 
[MRG+1] Relocate telnetconsole to extensions/
 2015-10-30 12:10:38 -03:00
Jakob de Maeyer 32ff4cc1d7 PEP8ify scrapy.settings tests 2015-10-29 19:52:19 +01:00
Jakob de Maeyer 51ca84c9b4 Add missing tests for scrapy.settings module 2015-10-29 17:34:18 +01:00
Mikhail Korobov caf2080b80 Merge pull request #1423 from jdemaeyer/enhancement/executionengine-close 
[MRG+1] Add ExecutionEngine.close() method
 2015-10-29 19:11:07 +05:00
Jakob de Maeyer 8307c1212f Add ExecutionEngine.close() method 2015-10-29 10:03:29 +01:00
Daniel Graña dd9f777ba7 Merge pull request #1149 from jdemaeyer/enhancement/settings-per-key-priorities 
[MRG+1] Per-key priorities for dict-like settings by promoting dicts to Settings instances
 2015-10-28 18:21:26 -03:00
Jakob de Maeyer 03f1720afb Fix backwards-compatibility for users who explicitly set _BASE settings 2015-10-27 14:22:56 +01:00
Jakob de Maeyer f249b309ab Move scrapy.utils.conf.remove_none_values to s.u.python.without_none_values 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 90198e5324 Add __repr__ method for BaseSettings 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 9eb3597d15 PEP8ify settings module 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 9bd7af8a62 Remove unused import in scrapy.settings 2015-10-27 12:43:33 +01:00
Jakob de Maeyer bb6dee611c Move Settings documentation to docstrings 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Jakob de Maeyer a769a1ef78 Introduce BaseSettings with full dictionary interface 2015-10-27 12:38:52 +01:00
Mikhail Korobov 4a9f76ebd3 Merge pull request #1555 from stummjr/patch-1 
fix typo in the documentation
 2015-10-26 12:30:46 +05:00
Valdir Stumm Jr d577c4702d fixed a typo in the documentation. 2015-10-26 00:00:20 -02:00
Pablo Hoffman 8dc400c66d Merge pull request #1541 from chripede/patch-1 
[MRG+1] Version 1 now exists
 2015-10-14 10:58:54 -03:00
Christian Pedersen cc76fb3dff Version 1 now exists 2015-10-14 14:07:34 +02:00
Pablo Hoffman 451318ef7a Merge pull request #1530 from scrapy/retry-400 
[MRG+1] DOC fix docs after GH-1289.
 2015-10-13 16:09:07 -03:00
Daniel Graña 411174cf38 Merge pull request #1535 from scrapy/py3-commands 
PY3 port bench, startproject, genspider, list and runspider commands
 2015-10-12 20:43:21 -03:00
Daniel Graña 65f4ba349c Merge pull request #1537 from scrapy/no-coveralls 
drop coveralls support
 2015-10-12 20:36:12 -03:00
Mikhail Korobov 09fc9b48b6 drop coveralls support 2015-10-12 18:12:56 +05:00
Mikhail Korobov 215905bdb6 PY3 port bench, startproject, genspider, list and runspider commands 2015-10-12 17:59:49 +05:00
Julia Medina 0117c811c9 Merge pull request #1533 from Digenis/xml_iter-nodename_with_dot 
[MRG+1] xmliter nodename with dot
 2015-10-12 09:33:27 -03:00
Mikhail Korobov 1b6d60c251 DOC fix docs after GH-1289. 2015-10-09 01:26:09 +05:00
Mikhail Korobov 9676cf8e53 Merge pull request #1527 from scrapy/no-pillow-300 
TST don't use broken Pillow version in tests
 2015-10-09 01:20:45 +05:00
Νικόλαος-Διγενής Καραγιάννης f56062d045 escape nodename in xmliter regex 2015-10-07 14:58:47 +03:00
Νικόλαος-Διγενής Καραγιάννης d66efb13ba test xml nodename with dots 2015-10-07 14:43:47 +03:00
Mikhail Korobov 4e9a43c228 Merge pull request #1532 from hoatle/docs-fixes 
Docs fixes
 2015-10-07 15:34:21 +05:00
hoatle 2869cf8dde fix another invalid xpath error 2015-10-07 16:03:43 +07:00
Hoat Le 4e66955411 fix ValueError: Invalid XPath: //div/[id="not-exists"]/text() on selectors.rst 
>>> response.xpath('//div/[id="not-exists"]/text()').extract_first() is None
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/http/response/text.py", line 109, in xpath
    return self.selector.xpath(query)
  File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/selector/unified.py", line 100, in xpath
    raise ValueError(msg if six.PY3 else msg.encode("unicode_escape"))
ValueError: Invalid XPath: //div/[id="not-exists"]/text()
 2015-10-07 15:43:02 +07:00
Mikhail Korobov 669be0a2b0 TST don't use broken Pillow version in tests 2015-10-06 17:24:52 +05:00
Νικόλαος-Διγενής Καραγιάννης d523c75f5e Relocate telnetconsole to extensions/ 2015-10-05 21:16:04 +03:00
Mikhail Korobov 91cbf97415 Merge pull request #1522 from smirecki/fix-typos-in-docs-topics 
DOC Typos corrections
 2015-10-03 13:23:38 +05:00
smirecki 8379bea7ed Typos corrections 
I've made a few small corrections, some spelling changes and typo fixes.
I've tried to respect regional spelling differences and avoided proposing hyphenating compound words.

 Please enter the commit message for your changes. Lines starting
 2015-10-02 23:48:27 -04:00
Daniel Graña fe15f93e53 Merge pull request #1498 from Preetwinder/scrapy-add_scheme 
[MRG+1] #1487 add_scheme_if_missing for `scrapy shell` command
 2015-09-30 21:38:58 -03:00
Marius Gedminas 0620e76433 Fix list formatting 2015-09-29 03:33:30 +05:00
Marius Gedminas eaad10facf Typo 2015-09-28 14:44:15 +05:00
Pablo Hoffman 75cd056223 disable log on version command. closes #1426 2015-09-25 16:39:21 -03:00
preetwinder 47c8e2ba78 Restructure tests for add_http_if_no_scheme function 2015-09-24 17:57:25 +00:00