Added fields to complete and mentors

Paul Tremberth 2014-02-06 13:42:04 -08:00
parent 7f78f6f6a6
commit e54b1cdf41
1 changed files with 105 additions and 1 deletions

@ -15,6 +15,14 @@ You can [join using the web browser](http://webchat.freenode.net/?channels=scrap
# Ideas for GSoC 2014
From http://en.flossmanuals.net/GSoCMentoring/making-your-ideas-page/
> an Ideas Page should have the following:
> * Brief descriptions of projects that can be completed in about 12 weeks.
> * For each project, a list of prerequisites, description of programming skills needed and estimation of difficulty level.
> * A list of potential mentors.
> In addition to a basic list, you might also consider providing links to relevant resources for mentors and students, particular FAQ entries, the timeline, etc. You might include a section on communication, giving specific advice on which mailing lists, channels and emails to use and how to use them.
Below is a list of potential projects for students. We've gathered these ideas from recent discussions on the mailing lists and among core developers.
This is for inspiration, you can apply with something completely different if you like. And if you do have another idea you'd like to propose, please consider sending it to http://groups.google.com/group/scrapy-developers for discussion.
@ -25,6 +33,12 @@ These ideas are intended to make Scrapy easier to use, increase its performance
### Better extension architecture and extensions repository
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Pablo Hoffman
Scrapy provides a lot of good hooks but makes it a bit complex
to enable extensions and to list which extensions are enabled,
partly because they are scattered around middlewares, pipelines, and bare extensions.
@ -44,6 +58,12 @@ which should encourage more writing more (and better) extensions
### Per-spider settings (SEP-19)
Brief explanation: add support for overriding settings per-spider
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Pablo Hoffman or Nicolas Ramirez
This is a proposal to add support for overriding settings per-spiders in a consistent way.
In short, you will be able to override settings (on a per-spider basis) by implementing a class method in your spider:
@ -66,6 +86,13 @@ In short, you will be able to override settings (on a per-spider basis) by imple
More details and some implementation proposals can be found in https://github.com/scrapy/scrapy/blob/master/sep/sep-019.rst
### Better generator support
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Mikhail Korobov
Mikhail:
> Integrating something like Rolando's https://github.com/darkrho/scrapy-inline-requests,
> writing more tests to ensure generators are not exhausted needlessly,
@ -73,6 +100,13 @@ Mikhail:
### Python 3 support
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Mikhail Korobov
Mikhail:
> Python 3 porting project is quite hard because Twisted doesn't even install on Mac with Python 3
> This project would require contributing not only Scrapy, but to Twisted as well.
@ -81,34 +115,82 @@ Shane:
> One good thing about this project is that it's easy to make progress and you can keep going.. I think it's OK if it doesn't go as far as windows or mac support.
### API from a Scrapy spider
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Shane Evans
Would take the same arguments as a Scrapy `Request` (url, callback,...)
and would return items and requests
### Better IPython integration
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Mikhail Korobov
(Mikhail, Shane)
Mikhail:
> develop IPython + Scrapy layer. It is possible to dislpay the HTML page inline in console, provide some interactive widgets and run Python code against the results (an old hacky demo of Scrapy+IPython is in attachements). IPython guys are going to release 2.0 version soon, and it should provide a standard protocol for such things.
### Profiling Scrapy
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Mikhail Korobov or Daniel Graña
Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure.
yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great.
### Improve JS integration
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Pablo Hoffman or Mikhail Korobov
Shane:
> maybe better scrapy-side support for splash or better scrapyjs integration
### Support for spiders other languages
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Shane Evans
Shane:
> maybe something like hadoop streaming where we have a spider that forks a process and manages communication with it.
### Firefox Addon to build spiders
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Shane Evans
Illarion:
> that helps to build spiders with complex logic visually,
> like defining steps to get list of all items and steps to perform for each item just by selecting and clicking parts of the webpage.
### Multi-platform Scrapy GUI for running spiders
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Rolando Espinoza
Rolando:
> would boost the choose of Scrapy for small projects by freelancers.
@ -126,15 +208,36 @@ The features could be:
Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.
### Replacing html regexp+python parsing with a c-based parser
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Shane Evans
Shane:
> e.g. gumbo parser, would need to research.
> The goal would be both performance and parsing quality.
### Better table support in scrapely
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Mikhail Korobov
Shane:
> (no need to annotate as much, or table-specific annotations)
### Support extracting fields in arbitrary order
Brief explanation: ---
Expected results: ---
Required skills: ---
Skill level: ---
Mentor: Shane Evans
Shane:
> (interesting, bit of research, testing, etc.), or just where we have tables (or similar) and the field name is in one column with the value in another (easier version of the problem)
@ -144,4 +247,5 @@ Shane:
* Shane Evans (@shane42)
* Pablo Hoffman (@pablohoffman)
* Mikhail Korobov (@kmike)
* Nicolas Ramirez (@nramirezuy) (possible)
* Nicolas Ramirez (@nramirezuy) (potential)
* Rolando Espinoza (@darkrho) (potential)