Updated GSoC 2014 (markdown)

Paul Tremberth 2014-02-06 10:41:49 -08:00
parent c857d0e620
commit 31509ef2b5
1 changed files with 36 additions and 1 deletions

@ -1 +1,36 @@
Scrapy is hoping to participate in Google Summer of Code in 2014. This page captures some ideas for student projects and provides information on how to get involved.
Scrapy is hoping to participate in Google Summer of Code in 2014. This page captures some ideas for student projects and provides information on how to get involved.
# Ideas for GSoC 2014
## Around Scrapy itself
### Better extension architecture and extensions repository
### Per-spider settings (SEP-19)
https://github.com/scrapy/scrapy/blob/master/sep/sep-019.rst
### Python 3 support
### API from a Scrapy spider
Would take the same arguments as a Scrapy `Request` (url, callback,...)
and would return items and requests
### Better IPython integration
(Mikhail, Shane)
### Profiling Scrapy
Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure.
yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great.
## Around Scrapely
### Replacing html regexp+python parsing with a c-based parser
e.g. gumbo parser, would need to research.
The goal would be both performance and parsing quality.
### Better table support in scrapely
(no need to annotate as much, or table-specific annotations)
### Support extracting fields in arbitrary order
(interesting, bit of research, testing, etc.), or just where we have tables (or similar) and the field name is in one column with the value in another (easier version of the problem)