diff --git a/GSoC-2014.md b/GSoC-2014.md index e23ed16..5f1e59e 100644 --- a/GSoC-2014.md +++ b/GSoC-2014.md @@ -1 +1,36 @@ -Scrapy is hoping to participate in Google Summer of Code in 2014. This page captures some ideas for student projects and provides information on how to get involved. \ No newline at end of file +Scrapy is hoping to participate in Google Summer of Code in 2014. This page captures some ideas for student projects and provides information on how to get involved. + +# Ideas for GSoC 2014 + +## Around Scrapy itself + +### Better extension architecture and extensions repository + +### Per-spider settings (SEP-19) +https://github.com/scrapy/scrapy/blob/master/sep/sep-019.rst + +### Python 3 support + +### API from a Scrapy spider +Would take the same arguments as a Scrapy `Request` (url, callback,...) +and would return items and requests + +### Better IPython integration +(Mikhail, Shane) + +### Profiling Scrapy +Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure. +yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great. + +## Around Scrapely + +### Replacing html regexp+python parsing with a c-based parser +e.g. gumbo parser, would need to research. +The goal would be both performance and parsing quality. + +### Better table support in scrapely +(no need to annotate as much, or table-specific annotations) + +### Support extracting fields in arbitrary order +(interesting, bit of research, testing, etc.), or just where we have tables (or similar) and the field name is in one column with the value in another (easier version of the problem) +