diff --git a/GSoC-2014.md b/GSoC-2014.md index 5f1e59e..45e4aec 100644 --- a/GSoC-2014.md +++ b/GSoC-2014.md @@ -6,11 +6,20 @@ Scrapy is hoping to participate in Google Summer of Code in 2014. This page cap ### Better extension architecture and extensions repository +Pablo to write a new SEP document. + ### Per-spider settings (SEP-19) https://github.com/scrapy/scrapy/blob/master/sep/sep-019.rst ### Python 3 support +Mikhail: +> Python 3 porting project is quite hard because Twisted doesn't even install on Mac with Python 3 +> This project would require contributing not only Scrapy, but to Twisted as well. + +Shane: +> One good thing about this project is that it's easy to make progress and you can keep going.. I think it's OK if it doesn't go as far as windows or mac support. + ### API from a Scrapy spider Would take the same arguments as a Scrapy `Request` (url, callback,...) and would return items and requests @@ -18,19 +27,51 @@ and would return items and requests ### Better IPython integration (Mikhail, Shane) +Mikhail: +> develop IPython + Scrapy layer. It is possible to dislpay the HTML page inline in console, provide some interactive widgets and run Python code against the results (an old hacky demo of Scrapy+IPython is in attachements). IPython guys are going to release 2.0 version soon, and it should provide a standard protocol for such things. + ### Profiling Scrapy Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure. yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great. +### Improve JS integration +Shane: +> maybe better scrapy-side support for splash or better scrapyjs integration + +### Support for spiders other languages +Shane: +> maybe something like hadoop streaming where we have a spider that forks a process and manages communication with it. + +### Firefox Addon to build spiders +Illarion: +> that helps to build spiders with complex logic visually, +> like defining steps to get list of all items and steps to perform for each item just by selecting and clicking parts of the webpage. + +### Multi-platform Scrapy GUI for running spiders + +Rolando: +> would boost the choose of Scrapy for small projects by freelancers. + +The features could be: + +* Can be started from scrapy command (i.e. scrapy startgui). Although it doesn't load the project code on gui loading, this allows to edit the spiders and don't need to restart the gui. +* Can manage jobs (start, stop, pause). +* Can help to set settings (like Dash, not modifying the actual settings.py). +* Can support obvios options (exporting to csv/jl, feed exporting to S3, stats email, etc). +* Can view/filter/search logs. +* Can view/search items. + ## Around Scrapely ### Replacing html regexp+python parsing with a c-based parser -e.g. gumbo parser, would need to research. -The goal would be both performance and parsing quality. +Shane: +> e.g. gumbo parser, would need to research. +> The goal would be both performance and parsing quality. ### Better table support in scrapely -(no need to annotate as much, or table-specific annotations) +Shane: +> (no need to annotate as much, or table-specific annotations) ### Support extracting fields in arbitrary order -(interesting, bit of research, testing, etc.), or just where we have tables (or similar) and the field name is in one column with the value in another (easier version of the problem) - +Shane: +> (interesting, bit of research, testing, etc.), or just where we have tables (or similar) and the field name is in one column with the value in another (easier version of the problem)