mirror of https://github.com/scrapy/scrapy.git
Updated GSoC 2014 (markdown)
parent
e54b1cdf41
commit
e84d1dd99e
154
GSoC-2014.md
154
GSoC-2014.md
|
|
@ -33,11 +33,15 @@ These ideas are intended to make Scrapy easier to use, increase its performance
|
|||
|
||||
### Better extension architecture and extensions repository
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Pablo Hoffman
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Pablo Hoffman
|
||||
|
||||
Scrapy provides a lot of good hooks but makes it a bit complex
|
||||
to enable extensions and to list which extensions are enabled,
|
||||
|
|
@ -58,11 +62,15 @@ which should encourage more writing more (and better) extensions
|
|||
|
||||
### Per-spider settings (SEP-19)
|
||||
|
||||
Brief explanation: add support for overriding settings per-spider
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Pablo Hoffman or Nicolas Ramirez
|
||||
**Brief explanation**: add support for overriding settings per-spider
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor** Pablo Hoffman or Nicolas Ramirez
|
||||
|
||||
This is a proposal to add support for overriding settings per-spiders in a consistent way.
|
||||
|
||||
|
|
@ -87,11 +95,15 @@ More details and some implementation proposals can be found in https://github.co
|
|||
|
||||
### Better generator support
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Mikhail Korobov
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Mikhail Korobov
|
||||
|
||||
Mikhail:
|
||||
> Integrating something like Rolando's https://github.com/darkrho/scrapy-inline-requests,
|
||||
|
|
@ -100,11 +112,15 @@ Mikhail:
|
|||
|
||||
### Python 3 support
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Mikhail Korobov
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Mikhail Korobov
|
||||
|
||||
|
||||
Mikhail:
|
||||
|
|
@ -116,22 +132,30 @@ Shane:
|
|||
|
||||
### API from a Scrapy spider
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Shane Evans
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Shane Evans
|
||||
|
||||
Would take the same arguments as a Scrapy `Request` (url, callback,...)
|
||||
and would return items and requests
|
||||
|
||||
### Better IPython integration
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Mikhail Korobov
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Mikhail Korobov
|
||||
|
||||
(Mikhail, Shane)
|
||||
|
||||
|
|
@ -140,44 +164,60 @@ Mikhail:
|
|||
|
||||
### Profiling Scrapy
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Mikhail Korobov or Daniel Graña
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Mikhail Korobov or Daniel Graña
|
||||
|
||||
Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure.
|
||||
yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great.
|
||||
|
||||
### Improve JS integration
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Pablo Hoffman or Mikhail Korobov
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Pablo Hoffman or Mikhail Korobov
|
||||
|
||||
Shane:
|
||||
> maybe better scrapy-side support for splash or better scrapyjs integration
|
||||
|
||||
### Support for spiders other languages
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Shane Evans
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Shane Evans
|
||||
|
||||
Shane:
|
||||
> maybe something like hadoop streaming where we have a spider that forks a process and manages communication with it.
|
||||
|
||||
### Firefox Addon to build spiders
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Shane Evans
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Shane Evans
|
||||
|
||||
Illarion:
|
||||
> that helps to build spiders with complex logic visually,
|
||||
|
|
@ -185,11 +225,15 @@ Illarion:
|
|||
|
||||
### Multi-platform Scrapy GUI for running spiders
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Rolando Espinoza
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Rolando Espinoza
|
||||
|
||||
Rolando:
|
||||
> would boost the choose of Scrapy for small projects by freelancers.
|
||||
|
|
|
|||
Loading…
Reference in New Issue