From 60d992535f0d810dd46c3bd90283281f7b45de28 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20Gra=C3=B1a?= Date: Mon, 10 Feb 2014 11:19:26 -0800 Subject: [PATCH] Updated GSoC 2014 (markdown) --- GSoC-2014.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/GSoC-2014.md b/GSoC-2014.md index aec65b5..af7057e 100644 --- a/GSoC-2014.md +++ b/GSoC-2014.md @@ -260,6 +260,18 @@ The features could be: **Mentor**: Daniel Graña +### New HTTP1.1 download handler + +**Brief explanation**: Replace current HTTP1.1 downloader handler with a in-house solution easily customizable to crawling needs. + +**Expected results**: It must [degrade nicely to parse invalid responses](https://github.com/scrapy/scrapy/issues/345), filtering out the [offending headers](https://github.com/scrapy/scrapy/issues/210) as browsers does. It must be able to [avoid downloading responses bigger than a size limit](https://github.com/scrapy/scrapy/issues/336), it can be configured to [throttle bandwidth used per download](https://github.com/scrapy/scrapy/issues/157), and if there is enough time it can lay out the interface to [response streaming](https://github.com/scrapy/scrapy/issues/440) + +**Required skills**: Python, Twisted and HTTP protocol + +**Skill level**: Medium + +**Mentor**: Daniel Graña + ## Around Scrapely Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.