RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated GSoC 2014 (markdown)

Daniel Graña 2014-02-10 10:20:22 -08:00
parent ed3b9dc7f3
commit 3ce0a4d813
1 changed files with 12 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
GSoC-2014.md

@ -248,6 +248,18 @@ The features could be:
* Can view/filter/search logs.
* Can view/search items.
### Integration tests
**Brief explanation**: Add integration tests for different networking scenarios
**Expected results**: Be able to tests from vertical to horizontal crawling against websites in same and different ips respecting throttling and handling timeouts, retries, dns failures. It must be simple to define new scenarios with predefined components (websites, proxies, routers, injected error rates)
**Required skills**: Python, Networking and Virtualization
**Skill level**: Medium
**Mentor**: Daniel Graña
## Around Scrapely
Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.