RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,393 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

10393 Commits

Author SHA1 Message Date
Jakob de Maeyer 9f7fcf5582 Make update_classpath() util function return non-string objects 2015-11-11 21:08:19 +01:00
Jakob de Maeyer d18b6a61d7 Add missing AddonManager tests 2015-11-11 21:08:19 +01:00
Jakob de Maeyer 18c7f3dbe2 Document built-in add-ons 2015-11-11 21:08:19 +01:00
Jakob de Maeyer 4ac6a83072 Fix class signatures in Extensions docs 2015-11-11 21:08:19 +01:00
Jakob de Maeyer 25498c3c21 Remove unused imports in add-ons 2015-11-11 21:08:19 +01:00
Jakob de Maeyer 2946b67414 Document add-ons 2015-11-11 21:08:19 +01:00
Jakob de Maeyer d91647c38b Add built-in add-ons 2015-11-11 21:08:19 +01:00
Jakob de Maeyer 07455b1883 Integrate add-ons into start-up process 2015-11-11 21:08:19 +01:00
Jakob de Maeyer d8af395d76 Introduce add-ons via AddonManager and Addon base class 2015-11-11 21:08:19 +01:00
Jakob de Maeyer e5b8def0b8 Redraft SEP-021 2015-11-11 21:08:19 +01:00
Jakob de Maeyer cfed9b6659 Allow passing Python objects to middleware dict settings 2015-11-11 21:08:19 +01:00
Julia Medina 54216d7afe Merge pull request #1586 from jdemaeyer/fix/backwards-compatible-per-key-priorities 
[MRG+1] Backwards compatible per key priorities
 2015-11-11 14:08:50 -03:00
Jakob de Maeyer 4f364764ae Simplify BaseSettings.__get__(), .getpriority() 2015-11-11 17:40:42 +01:00
Jakob de Maeyer 44f6ada034 Overwrite, not update, dictionary-like settings 2015-11-11 17:40:42 +01:00
Jakob de Maeyer 52ecee6a62 Replace BaseSettings._getcomposite() with public .getwithbase() method 2015-11-11 17:39:57 +01:00
Jakob de Maeyer b6a023ce98 Add backwards compatibility for build_component_list 2015-11-11 17:39:56 +01:00
Jakob de Maeyer e66f649894 Bring back _BASE settings 2015-11-11 17:39:56 +01:00
Jakob de Maeyer 2629997a2f Make Spider.update_settings() an instance method 2015-11-10 23:48:20 +01:00
Mikhail Korobov 9548691fdd Merge pull request #1563 from starrify/master 
[MRG+1] fixed: Issue #1562 (Incorrectly picked URL in `scrapy.http.FormReques t.from_response` when there is a `<base>` tag)
 2015-11-10 21:33:24 +05:00
Jakob de Maeyer daec045711 Move spider settings tests 2015-11-09 18:07:05 +01:00
Jakob de Maeyer 380f76d35f Fix tests that had multiple calls to crawl() 2015-11-09 16:54:22 +01:00
Jakob de Maeyer fc26397319 Revert "Allow multiple calls to Crawler.crawl()" 
This reverts commit 4e40377bcb.
 2015-11-09 16:32:45 +01:00
Jakob de Maeyer aafb31d6fb Revert "Move stats & log init into crawl()" 
This reverts commit 2c68c95cad.
 2015-11-09 16:26:55 +01:00
Jakob de Maeyer 2c68c95cad Move stats & log init into crawl() 2015-11-03 23:46:48 +01:00
Jakob de Maeyer 4e40377bcb Allow multiple calls to Crawler.crawl() 2015-11-03 23:32:02 +01:00
Leonid Amirov 97b51ea33b issue GH #1550 - six library is used instead of urlparse for python3 compatibility 2015-11-03 14:57:37 +03:00
Leonid Amirov dd45b31fe4 issue GH #1550 - rewritten add_http_if_no_scheme() 2015-11-03 14:32:30 +03:00
Jakob de Maeyer 86c74ce53e Allow Spider.update_settings() to be an instance method 2015-11-02 16:57:57 +01:00
Jakob de Maeyer b06a670777 Initialize spider before calling its update_settings() 2015-11-02 16:47:07 +01:00
Jakob de Maeyer d67f292d92 Move Spider.update_settings() into Crawler.crawl() 2015-11-02 16:36:33 +01:00
Jakob de Maeyer c0566b2b07 Move extension init into Crawler.crawl() 2015-11-02 16:18:56 +01:00
Pengyu CHEN 94486bb294 added: Test case for the fix. 2015-11-02 23:00:42 +08:00
Pengyu CHEN e19bf4aecc added: Test case for the fix 2015-11-02 22:52:41 +08:00
Leonid Amirov bc9db65358 issue GH #1550 - scrapy shell argument fixes: "example.com" requests "http://example.com"; "example" requests "file://example"; "./example.com" requests "file://example.com" 2015-11-02 16:08:19 +03:00
Leonid Amirov a41c64bfb9 issue GH #1550 - fixed bugs in scrapy.utils.url.add_http_if_no_scheme(): when given URI where scheme is present, but not 'http' the function gave bad result 2015-11-02 16:06:21 +03:00
Leonid Amirov 98a2e77a75 issue GH #1550 - fixed error: shell command wasn't accepting files URIs 2015-11-02 15:30:49 +03:00
Julia Medina 57f87b95d4 Merge pull request #1528 from scrapy/create-crawler 
public Crawler.create_crawler method
 2015-10-30 12:36:35 -03:00
Julia Medina 9424ca0fdb Merge pull request #1570 from jdemaeyer/tests/complete-settings-tests 
[MRG+1] Add some missing tests for scrapy.settings
 2015-10-30 12:36:25 -03:00
Mikhail Korobov 0000b6e9ef TST cleanup scrapy.utils.test.get_crawler 2015-10-30 20:27:03 +05:00
Mikhail Korobov a49c82ad62 TST improve CrawlerRunner tests 
* use CrawlerRunner.create_crawler instead of get_crawler helper in test_crawl;
* add a test for loading spiders by name;
* add a test for passing Crawler objects instead of Spider objects;
* add a test for CrawlerRunner.join
 2015-10-30 20:24:37 +05:00
Mikhail Korobov 11b11c9803 CrawlerRunner.create_crawler method 2015-10-30 20:22:31 +05:00
Julia Medina 72eeead6db Merge pull request #1524 from Digenis/relocate_telnet_console 
[MRG+1] Relocate telnetconsole to extensions/
 2015-10-30 12:10:38 -03:00
Jakob de Maeyer 32ff4cc1d7 PEP8ify scrapy.settings tests 2015-10-29 19:52:19 +01:00
Jakob de Maeyer 51ca84c9b4 Add missing tests for scrapy.settings module 2015-10-29 17:34:18 +01:00
Mikhail Korobov caf2080b80 Merge pull request #1423 from jdemaeyer/enhancement/executionengine-close 
[MRG+1] Add ExecutionEngine.close() method
 2015-10-29 19:11:07 +05:00
Jakob de Maeyer 8307c1212f Add ExecutionEngine.close() method 2015-10-29 10:03:29 +01:00
Pengyu CHEN e379f58cad fixed: Issue #1564 (Incorrectly picked URL in `scrapy.linkextractors.regex.RegexLinkExtractor` when there is a `<base>` tag. ) 2015-10-29 14:52:31 +08:00
Pengyu CHEN c34dbe955d fixed: Issue #1562 (Incorrectly picked URL in `scrapy.http.FormRequest.from_response` when there is a `<base>` tag) 2015-10-29 14:18:59 +08:00
Daniel Graña dd9f777ba7 Merge pull request #1149 from jdemaeyer/enhancement/settings-per-key-priorities 
[MRG+1] Per-key priorities for dict-like settings by promoting dicts to Settings instances
 2015-10-28 18:21:26 -03:00
Jakob de Maeyer 03f1720afb Fix backwards-compatibility for users who explicitly set _BASE settings 2015-10-27 14:22:56 +01:00