mirror of https://github.com/scrapy/scrapy.git
Added fields to complete and mentors
parent
7f78f6f6a6
commit
e54b1cdf41
106
GSoC-2014.md
106
GSoC-2014.md
|
|
@ -15,6 +15,14 @@ You can [join using the web browser](http://webchat.freenode.net/?channels=scrap
|
|||
|
||||
# Ideas for GSoC 2014
|
||||
|
||||
From http://en.flossmanuals.net/GSoCMentoring/making-your-ideas-page/
|
||||
|
||||
> an Ideas Page should have the following:
|
||||
> * Brief descriptions of projects that can be completed in about 12 weeks.
|
||||
> * For each project, a list of prerequisites, description of programming skills needed and estimation of difficulty level.
|
||||
> * A list of potential mentors.
|
||||
> In addition to a basic list, you might also consider providing links to relevant resources for mentors and students, particular FAQ entries, the timeline, etc. You might include a section on communication, giving specific advice on which mailing lists, channels and emails to use and how to use them.
|
||||
|
||||
Below is a list of potential projects for students. We've gathered these ideas from recent discussions on the mailing lists and among core developers.
|
||||
|
||||
This is for inspiration, you can apply with something completely different if you like. And if you do have another idea you'd like to propose, please consider sending it to http://groups.google.com/group/scrapy-developers for discussion.
|
||||
|
|
@ -25,6 +33,12 @@ These ideas are intended to make Scrapy easier to use, increase its performance
|
|||
|
||||
### Better extension architecture and extensions repository
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Pablo Hoffman
|
||||
|
||||
Scrapy provides a lot of good hooks but makes it a bit complex
|
||||
to enable extensions and to list which extensions are enabled,
|
||||
partly because they are scattered around middlewares, pipelines, and bare extensions.
|
||||
|
|
@ -44,6 +58,12 @@ which should encourage more writing more (and better) extensions
|
|||
|
||||
### Per-spider settings (SEP-19)
|
||||
|
||||
Brief explanation: add support for overriding settings per-spider
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Pablo Hoffman or Nicolas Ramirez
|
||||
|
||||
This is a proposal to add support for overriding settings per-spiders in a consistent way.
|
||||
|
||||
In short, you will be able to override settings (on a per-spider basis) by implementing a class method in your spider:
|
||||
|
|
@ -66,6 +86,13 @@ In short, you will be able to override settings (on a per-spider basis) by imple
|
|||
More details and some implementation proposals can be found in https://github.com/scrapy/scrapy/blob/master/sep/sep-019.rst
|
||||
|
||||
### Better generator support
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Mikhail Korobov
|
||||
|
||||
Mikhail:
|
||||
> Integrating something like Rolando's https://github.com/darkrho/scrapy-inline-requests,
|
||||
> writing more tests to ensure generators are not exhausted needlessly,
|
||||
|
|
@ -73,6 +100,13 @@ Mikhail:
|
|||
|
||||
### Python 3 support
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Mikhail Korobov
|
||||
|
||||
|
||||
Mikhail:
|
||||
> Python 3 porting project is quite hard because Twisted doesn't even install on Mac with Python 3
|
||||
> This project would require contributing not only Scrapy, but to Twisted as well.
|
||||
|
|
@ -81,34 +115,82 @@ Shane:
|
|||
> One good thing about this project is that it's easy to make progress and you can keep going.. I think it's OK if it doesn't go as far as windows or mac support.
|
||||
|
||||
### API from a Scrapy spider
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Shane Evans
|
||||
|
||||
Would take the same arguments as a Scrapy `Request` (url, callback,...)
|
||||
and would return items and requests
|
||||
|
||||
### Better IPython integration
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Mikhail Korobov
|
||||
|
||||
(Mikhail, Shane)
|
||||
|
||||
Mikhail:
|
||||
> develop IPython + Scrapy layer. It is possible to dislpay the HTML page inline in console, provide some interactive widgets and run Python code against the results (an old hacky demo of Scrapy+IPython is in attachements). IPython guys are going to release 2.0 version soon, and it should provide a standard protocol for such things.
|
||||
|
||||
### Profiling Scrapy
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Mikhail Korobov or Daniel Graña
|
||||
|
||||
Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure.
|
||||
yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great.
|
||||
|
||||
### Improve JS integration
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Pablo Hoffman or Mikhail Korobov
|
||||
|
||||
Shane:
|
||||
> maybe better scrapy-side support for splash or better scrapyjs integration
|
||||
|
||||
### Support for spiders other languages
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Shane Evans
|
||||
|
||||
Shane:
|
||||
> maybe something like hadoop streaming where we have a spider that forks a process and manages communication with it.
|
||||
|
||||
### Firefox Addon to build spiders
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Shane Evans
|
||||
|
||||
Illarion:
|
||||
> that helps to build spiders with complex logic visually,
|
||||
> like defining steps to get list of all items and steps to perform for each item just by selecting and clicking parts of the webpage.
|
||||
|
||||
### Multi-platform Scrapy GUI for running spiders
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Rolando Espinoza
|
||||
|
||||
Rolando:
|
||||
> would boost the choose of Scrapy for small projects by freelancers.
|
||||
|
||||
|
|
@ -126,15 +208,36 @@ The features could be:
|
|||
Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.
|
||||
|
||||
### Replacing html regexp+python parsing with a c-based parser
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Shane Evans
|
||||
|
||||
Shane:
|
||||
> e.g. gumbo parser, would need to research.
|
||||
> The goal would be both performance and parsing quality.
|
||||
|
||||
### Better table support in scrapely
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Mikhail Korobov
|
||||
|
||||
Shane:
|
||||
> (no need to annotate as much, or table-specific annotations)
|
||||
|
||||
### Support extracting fields in arbitrary order
|
||||
|
||||
Brief explanation: ---
|
||||
Expected results: ---
|
||||
Required skills: ---
|
||||
Skill level: ---
|
||||
Mentor: Shane Evans
|
||||
|
||||
Shane:
|
||||
> (interesting, bit of research, testing, etc.), or just where we have tables (or similar) and the field name is in one column with the value in another (easier version of the problem)
|
||||
|
||||
|
|
@ -144,4 +247,5 @@ Shane:
|
|||
* Shane Evans (@shane42)
|
||||
* Pablo Hoffman (@pablohoffman)
|
||||
* Mikhail Korobov (@kmike)
|
||||
* Nicolas Ramirez (@nramirezuy) (possible)
|
||||
* Nicolas Ramirez (@nramirezuy) (potential)
|
||||
* Rolando Espinoza (@darkrho) (potential)
|
||||
Loading…
Reference in New Issue