RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated GSoC 2014 (markdown)

Daniel Graña 2014-02-10 11:19:26 -08:00
parent 3ce0a4d813
commit 60d992535f
1 changed files with 12 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
GSoC-2014.md

@ -260,6 +260,18 @@ The features could be:
**Mentor**: Daniel Graña
### New HTTP1.1 download handler
**Brief explanation**: Replace current HTTP1.1 downloader handler with a in-house solution easily customizable to crawling needs.
**Expected results**: It must [degrade nicely to parse invalid responses](https://github.com/scrapy/scrapy/issues/345), filtering out the [offending headers](https://github.com/scrapy/scrapy/issues/210) as browsers does. It must be able to [avoid downloading responses bigger than a size limit](https://github.com/scrapy/scrapy/issues/336), it can be configured to [throttle bandwidth used per download](https://github.com/scrapy/scrapy/issues/157), and if there is enough time it can lay out the interface to [response streaming](https://github.com/scrapy/scrapy/issues/440)
**Required skills**: Python, Twisted and HTTP protocol
**Skill level**: Medium
**Mentor**: Daniel Graña
## Around Scrapely
Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.