RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated GSoC 2014 (markdown)

Daniel Graña 2014-02-10 11:26:40 -08:00
parent 60d992535f
commit 02e090ee9b
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
GSoC-2014.md

@ -272,6 +272,10 @@ The features could be:
**Mentor**: Daniel Graña
Current HTTP1.1 download handler depends on code shipped with Twisted that is not easily extensible by us, we ship twisted code under `scrapy.xlib.tx` to support running Scrapy in older twisted versions for distributions that doesn't ship uptodate Twisted packages. But this is an ongoing cat-mouse game, the http download handler is an essential component of a crawling framework and having no control over its release cycle leaves us with code that is hard to support.
The idea of this task is to depart from current Twisted code looking for a design that can cover current and future needs taking in count the goal is to deal with websites that doesn't follow standards to the letter.
## Around Scrapely
Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.