From 3ce0a4d8137e663d593aa9ee64af9c29e7ee07af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20Gra=C3=B1a?= Date: Mon, 10 Feb 2014 10:20:22 -0800 Subject: [PATCH] Updated GSoC 2014 (markdown) --- GSoC-2014.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/GSoC-2014.md b/GSoC-2014.md index d9f0e03..aec65b5 100644 --- a/GSoC-2014.md +++ b/GSoC-2014.md @@ -248,6 +248,18 @@ The features could be: * Can view/filter/search logs. * Can view/search items. +### Integration tests + +**Brief explanation**: Add integration tests for different networking scenarios + +**Expected results**: Be able to tests from vertical to horizontal crawling against websites in same and different ips respecting throttling and handling timeouts, retries, dns failures. It must be simple to define new scenarios with predefined components (websites, proxies, routers, injected error rates) + +**Required skills**: Python, Networking and Virtualization + +**Skill level**: Medium + +**Mentor**: Daniel GraƱa + ## Around Scrapely Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.