diff --git a/GSoC-2014.md b/GSoC-2014.md index af7057e..8d9b296 100644 --- a/GSoC-2014.md +++ b/GSoC-2014.md @@ -272,6 +272,10 @@ The features could be: **Mentor**: Daniel GraƱa +Current HTTP1.1 download handler depends on code shipped with Twisted that is not easily extensible by us, we ship twisted code under `scrapy.xlib.tx` to support running Scrapy in older twisted versions for distributions that doesn't ship uptodate Twisted packages. But this is an ongoing cat-mouse game, the http download handler is an essential component of a crawling framework and having no control over its release cycle leaves us with code that is hard to support. + +The idea of this task is to depart from current Twisted code looking for a design that can cover current and future needs taking in count the goal is to deal with websites that doesn't follow standards to the letter. + ## Around Scrapely Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.