From e84d1dd99e6a0f776f65446f37d752222cb0622f Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Thu, 6 Feb 2014 13:44:20 -0800 Subject: [PATCH] Updated GSoC 2014 (markdown) --- GSoC-2014.md | 154 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 99 insertions(+), 55 deletions(-) diff --git a/GSoC-2014.md b/GSoC-2014.md index dcb687e..197b99d 100644 --- a/GSoC-2014.md +++ b/GSoC-2014.md @@ -33,11 +33,15 @@ These ideas are intended to make Scrapy easier to use, increase its performance ### Better extension architecture and extensions repository -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Pablo Hoffman +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Pablo Hoffman Scrapy provides a lot of good hooks but makes it a bit complex to enable extensions and to list which extensions are enabled, @@ -58,11 +62,15 @@ which should encourage more writing more (and better) extensions ### Per-spider settings (SEP-19) -Brief explanation: add support for overriding settings per-spider -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Pablo Hoffman or Nicolas Ramirez +**Brief explanation**: add support for overriding settings per-spider + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor** Pablo Hoffman or Nicolas Ramirez This is a proposal to add support for overriding settings per-spiders in a consistent way. @@ -87,11 +95,15 @@ More details and some implementation proposals can be found in https://github.co ### Better generator support -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Mikhail Korobov +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Mikhail Korobov Mikhail: > Integrating something like Rolando's https://github.com/darkrho/scrapy-inline-requests, @@ -100,11 +112,15 @@ Mikhail: ### Python 3 support -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Mikhail Korobov +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Mikhail Korobov Mikhail: @@ -116,22 +132,30 @@ Shane: ### API from a Scrapy spider -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Shane Evans +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Shane Evans Would take the same arguments as a Scrapy `Request` (url, callback,...) and would return items and requests ### Better IPython integration -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Mikhail Korobov +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Mikhail Korobov (Mikhail, Shane) @@ -140,44 +164,60 @@ Mikhail: ### Profiling Scrapy -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Mikhail Korobov or Daniel Graña +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Mikhail Korobov or Daniel Graña Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure. yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great. ### Improve JS integration -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Pablo Hoffman or Mikhail Korobov +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Pablo Hoffman or Mikhail Korobov Shane: > maybe better scrapy-side support for splash or better scrapyjs integration ### Support for spiders other languages -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Shane Evans +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Shane Evans Shane: > maybe something like hadoop streaming where we have a spider that forks a process and manages communication with it. ### Firefox Addon to build spiders -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Shane Evans +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Shane Evans Illarion: > that helps to build spiders with complex logic visually, @@ -185,11 +225,15 @@ Illarion: ### Multi-platform Scrapy GUI for running spiders -Brief explanation: --- -Expected results: --- -Required skills: --- -Skill level: --- -Mentor: Rolando Espinoza +**Brief explanation**: --- + +**Expected results**: --- + +**Required skills**: --- + +**Skill level**: --- + +**Mentor**: Rolando Espinoza Rolando: > would boost the choose of Scrapy for small projects by freelancers.