diff --git a/GSoC-2014.md b/GSoC-2014.md index aec65b5..af7057e 100644 --- a/GSoC-2014.md +++ b/GSoC-2014.md @@ -260,6 +260,18 @@ The features could be: **Mentor**: Daniel Graña +### New HTTP1.1 download handler + +**Brief explanation**: Replace current HTTP1.1 downloader handler with a in-house solution easily customizable to crawling needs. + +**Expected results**: It must [degrade nicely to parse invalid responses](https://github.com/scrapy/scrapy/issues/345), filtering out the [offending headers](https://github.com/scrapy/scrapy/issues/210) as browsers does. It must be able to [avoid downloading responses bigger than a size limit](https://github.com/scrapy/scrapy/issues/336), it can be configured to [throttle bandwidth used per download](https://github.com/scrapy/scrapy/issues/157), and if there is enough time it can lay out the interface to [response streaming](https://github.com/scrapy/scrapy/issues/440) + +**Required skills**: Python, Twisted and HTTP protocol + +**Skill level**: Medium + +**Mentor**: Daniel Graña + ## Around Scrapely Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.