RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,549 Commits 42 Branches 132 Tags 161 MiB
Commit Graph

3197 Commits

Author SHA1 Message Date
nyov 6d48c19acd dissolve `scrapy.contrib_exp` 2015-04-07 21:20:48 +00:00
Pablo Hoffman 4b11501eb5 Merge pull request #963 from tpeng/fix-xmliter-lxml 
[MRG+1] support namespace prefix in xmliter_lxml
 2015-04-07 18:13:01 -03:00
Daniel Graña 9706119336 Merge pull request #1128 from Curita/move-spider-settings-population-to-crawler 
[MRG+1] Move spider settings population from CrawlerRunner to Crawler.__init__
 2015-04-04 18:08:05 -03:00
Julia Medina 6f9265b44b Move spider settings population to Crawler.__init__ 2015-04-01 23:29:10 -03:00
nyov dc88be75e2 more python3 compatibility changes for urllib 2015-04-01 19:50:56 +00:00
Daniel Graña 27591b55fc Merge pull request #1123 from sibiryakov/reactor-threadpool-size 
[MRG+1] Reactor threadpool max size setting
 2015-04-01 15:08:03 -03:00
Alexander Sibiryakov 5916df64cd Removing unnecessary import. 2015-04-01 11:24:12 +02:00
Daniel Graña ec4251afd7 Merge pull request #1089 from drack3800/master 
[MRG+1] Add test for webclient with POST method and no body given
 2015-03-31 17:30:08 -03:00
Alexander Sibiryakov c1cd019f56 Setting maximum thread pool size in reactor. 2015-03-31 10:56:31 +02:00
Nicolás Alejandro Ramírez Quiros a3702e749f Merge pull request #1073 from eliasdorneles/add-more-settings-to-template 
[MRG+1] Adding more settings to project template
 2015-03-27 15:37:26 -03:00
Pablo Hoffman bb4c922d85 Merge pull request #1081 from scrapy/dict-items 
Allow spiders to return dicts.
 2015-03-27 15:19:27 -03:00
Daniel Graña 55a23d102f Merge pull request #1086 from Curita/response-urljoin 
Add Response.urljoin() helper
 2015-03-27 15:17:54 -03:00
nyov 84edc2ebc7 Add Response.urljoin() testcase 
and add evaluation of base-url for HtmlResponse.
 2015-03-27 05:59:01 +00:00
Mikhail Korobov 39085ae18f Merge pull request #1098 from nyov/nyov/userconfig 
[+1 MRG]look in ~/.config/scrapy.cfg for user config
 2015-03-27 02:11:35 +05:00
nyov 1134a9cab0 config: look in ~/.config/scrapy.cfg as well 2015-03-26 20:36:14 +00:00
drack3800 6afd1a78bd style fix 2015-03-25 20:43:07 +03:00
nyov aaeb837db4 handle TLS SNI if we have twisted>=14.0 
(closes #981, #1101)
 2015-03-24 10:52:29 +00:00
drack3800 549882590b Fixed bug with no specified Content-Length header by ScrapyHTTPClientFactory for POST request with no given body 2015-03-22 19:31:29 +03:00
drack3800 deb5bb530c Fixed bug with no specified Content-Length header by ScrapyHTTPClientFactory for POST request with no given body 2015-03-22 19:25:08 +03:00
Julia Medina cda3922507 Add Response.urljoin() helper 2015-03-19 19:07:52 -03:00
Elias Dorneles d7cb2b9a91 making commented code indentation consistent 2015-03-19 15:41:43 -03:00
Julia Medina 0dade7315b Use generator sintax in re_first 2015-03-18 21:11:08 -03:00
Mateusz Golewski f92bc09bf4 Add re_first() to SelectorList and iflatten() to utils.python 2015-03-18 21:11:08 -03:00
Mateusz Golewski bd126be356 Optimize extract_first() 2015-03-18 21:11:08 -03:00
Mateusz Golewski 6c7bd54fc3 Add extract_first() method to SelectorList 2015-03-18 21:11:08 -03:00
Nicolás Alejandro Ramírez Quiros ee82fe0e24 Merge pull request #1016 from SudShekhar/jsonProcessor 
[MRG+1] Added JmesSelect
 2015-03-18 08:11:25 -03:00
Pablo Hoffman 5846d6154c emphasize web crawling over screen scraping on scrapy description. closes #586 2015-03-18 00:24:16 -03:00
Mikhail Korobov 39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Mikhail Korobov da90449edf typo fix in scrapy.contrib.pipeline.media 2015-03-18 07:24:15 +05:00
Mikhail Korobov d14c972ed9 Merge pull request #1079 from kmike/feed-iterate-spider-output 
CSVFeedSpider cleanup: use iterate_spider_output
 2015-03-18 04:44:50 +05:00
Pablo Hoffman 3f15b6df8b Merge pull request #1078 from kmike/iter-spider-output 
remove unnecessary check from scrapy.utils.spider.iter_spider_output
 2015-03-17 16:36:43 -03:00
Mikhail Korobov 6974902323 CSVFeedSpider cleanup: use iterate_spider_output 
A similar fix was made for XMLFeedSpider in 95fde0a498
 2015-03-17 23:31:29 +05:00
Mikhail Korobov 8ce4ad0615 remove unnecessary check from scrapy.utils.spider.iter_spider_output 
arg_to_iter handles Items since 2bbd92742b
 2015-03-17 23:07:39 +05:00
Mikhail Korobov 62c4481d78 Merge pull request #992 from chekunkov/pydispatch_pep8 
Pydispatch pep8
 2015-03-17 22:42:31 +05:00
Pablo Hoffman 934584a355 Merge pull request #1020 from jojje/gzip_http_cache 
[MRG+1] add gzip compression to filesystem http cache backend
 2015-03-17 14:32:06 -03:00
Elias Dorneles fd67fe273a using default values for settings that are off by default 2015-03-17 09:34:54 -03:00
Pablo Hoffman f924567591 Merge pull request #983 from ArturGaspar/linkextractor_css 
[MRG+1] CSS support in link extractors
 2015-03-17 01:07:47 -03:00
Alexander Chekunkov 05cb31d306 pydispatch - pep8 
reformat files - proper spacing
 2015-03-16 21:43:43 +00:00
Elias Dorneles aa56dd30ea updating new settings to template, as pointed in PR review 2015-03-16 17:36:46 -03:00
nramirezuy c13e23641b httpcache dont_cache meta #19 #689 2015-03-16 11:50:04 -03:00
Elias Dorneles 05a88152af adding more settings to project template 2015-03-14 16:12:37 -03:00
Sudhanshu Shekhar 839ffba971 Added the first version of SelectJmes 
Utilizes jmespath. Also, added tests and documentation for the same.
 2015-02-24 22:59:01 +05:30
Julia Medina d67ca77e61 Support new _getEndpoint Agent signatures on Twisted 15.0.0 2015-02-15 21:14:47 -03:00
Maik Röder 721d8d5dae Update form.py 
Typo fixed
 2015-02-04 11:38:37 +01:00
Jonas Tingeborn bd5d99a2d2 add gzip compression to filesystem http cache backend 2015-01-21 20:18:11 +01:00
Julia Medina d68615a5af Test the parse command locally instead of against an external url 2015-01-19 10:28:25 -03:00
Julia Medina 09ba4ff68a Patches Twisted issue while closing the connection pool on HTTPDownloadHandler 2014-12-30 19:53:50 -03:00
nyov 880c9e52b5 t.i.b.ThreadedResolver is now a new-style class 
since 2009 / twisted-9.0.0
663d669dce
 2014-12-16 19:28:39 +00:00
tpeng 82d138e87e support namespace prefix in xmliter_lxml 2014-12-15 16:17:06 +01:00
Artur Gaspar 22247cf791 move restrict_css argument to end of argument list in link extractors for backwards compatibility, use keyword arguments in link extractor super().__init__() calls 2014-12-15 09:18:15 -02:00