mirror of https://github.com/scrapy/scrapy.git
Updated GSoC 2014 (markdown)
parent
c857d0e620
commit
31509ef2b5
37
GSoC-2014.md
37
GSoC-2014.md
|
|
@ -1 +1,36 @@
|
|||
Scrapy is hoping to participate in Google Summer of Code in 2014. This page captures some ideas for student projects and provides information on how to get involved.
|
||||
Scrapy is hoping to participate in Google Summer of Code in 2014. This page captures some ideas for student projects and provides information on how to get involved.
|
||||
|
||||
# Ideas for GSoC 2014
|
||||
|
||||
## Around Scrapy itself
|
||||
|
||||
### Better extension architecture and extensions repository
|
||||
|
||||
### Per-spider settings (SEP-19)
|
||||
https://github.com/scrapy/scrapy/blob/master/sep/sep-019.rst
|
||||
|
||||
### Python 3 support
|
||||
|
||||
### API from a Scrapy spider
|
||||
Would take the same arguments as a Scrapy `Request` (url, callback,...)
|
||||
and would return items and requests
|
||||
|
||||
### Better IPython integration
|
||||
(Mikhail, Shane)
|
||||
|
||||
### Profiling Scrapy
|
||||
Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure.
|
||||
yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great.
|
||||
|
||||
## Around Scrapely
|
||||
|
||||
### Replacing html regexp+python parsing with a c-based parser
|
||||
e.g. gumbo parser, would need to research.
|
||||
The goal would be both performance and parsing quality.
|
||||
|
||||
### Better table support in scrapely
|
||||
(no need to annotate as much, or table-specific annotations)
|
||||
|
||||
### Support extracting fields in arbitrary order
|
||||
(interesting, bit of research, testing, etc.), or just where we have tables (or similar) and the field name is in one column with the value in another (easier version of the problem)
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue