Updated GSoC 2014 (markdown)

Paul Tremberth 2014-02-06 13:44:20 -08:00
parent e54b1cdf41
commit e84d1dd99e
1 changed files with 99 additions and 55 deletions

@ -33,11 +33,15 @@ These ideas are intended to make Scrapy easier to use, increase its performance
### Better extension architecture and extensions repository
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Pablo Hoffman
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Pablo Hoffman
Scrapy provides a lot of good hooks but makes it a bit complex
to enable extensions and to list which extensions are enabled,
@ -58,11 +62,15 @@ which should encourage more writing more (and better) extensions
### Per-spider settings (SEP-19)
Brief explanation: add support for overriding settings per-spider
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Pablo Hoffman or Nicolas Ramirez
**Brief explanation**: add support for overriding settings per-spider
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor** Pablo Hoffman or Nicolas Ramirez
This is a proposal to add support for overriding settings per-spiders in a consistent way.
@ -87,11 +95,15 @@ More details and some implementation proposals can be found in https://github.co
### Better generator support
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Mikhail Korobov
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Mikhail Korobov
Mikhail:
> Integrating something like Rolando's https://github.com/darkrho/scrapy-inline-requests,
@ -100,11 +112,15 @@ Mikhail:
### Python 3 support
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Mikhail Korobov
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Mikhail Korobov
Mikhail:
@ -116,22 +132,30 @@ Shane:
### API from a Scrapy spider
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Shane Evans
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Shane Evans
Would take the same arguments as a Scrapy `Request` (url, callback,...)
and would return items and requests
### Better IPython integration
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Mikhail Korobov
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Mikhail Korobov
(Mikhail, Shane)
@ -140,44 +164,60 @@ Mikhail:
### Profiling Scrapy
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Mikhail Korobov or Daniel Graña
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Mikhail Korobov or Daniel Graña
Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure.
yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great.
### Improve JS integration
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Pablo Hoffman or Mikhail Korobov
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Pablo Hoffman or Mikhail Korobov
Shane:
> maybe better scrapy-side support for splash or better scrapyjs integration
### Support for spiders other languages
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Shane Evans
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Shane Evans
Shane:
> maybe something like hadoop streaming where we have a spider that forks a process and manages communication with it.
### Firefox Addon to build spiders
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Shane Evans
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Shane Evans
Illarion:
> that helps to build spiders with complex logic visually,
@ -185,11 +225,15 @@ Illarion:
### Multi-platform Scrapy GUI for running spiders
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Rolando Espinoza
**Brief explanation**: ---
**Expected results**: ---
**Required skills**: ---
**Skill level**: ---
**Mentor**: Rolando Espinoza
Rolando:
> would boost the choose of Scrapy for small projects by freelancers.