RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

renamed draft ideas page and moved 'better generator support' there

Pablo Hoffman 2014-02-14 14:05:44 -02:00
parent 36bdc6b4c1
commit ff13afdb7e
4 changed files with 42 additions and 41 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
GSoC-2014-Draft-Ideas.rst Normal file

@ -0,0 +1,40 @@
Google Summer of Code - Draft Ideas
===================================
Better generator support
------------------------
TODO:
This one should be better specified.
==================== ===========
Brief explanation Improve Scrapy API using generators
Expected results Scrapy should provide an easy way to build a single item from several pages, ...
Required skills Python, general understanding of async code, API design
Mentor(s) Mikhail Korobov, Rolando Espinoza
==================== ===========
There are areas where Scrapy usability and efficiency can be improved by using generators, for example:
* Integrate something like Rolando's https://github.com/darkrho/scrapy-inline-requests;
* ensure generators are not exhausted needlessly in various places;
* provide an easier alternative to spider_idle signal, something in line with https://github.com/scrapy/scrapy/issues/456
* ...
Reading list:
* http://www.python.org/dev/peps/pep-0342/
* http://www.tornadoweb.org/en/stable/gen.html
* http://twistedmatrix.com/documents/13.0.0/core/howto/defer.html
* https://twistedmatrix.com/trac/wiki/DeferredGenerator
Improve javascript integration
------------------------------
==================== ===========
Brief explanation Improve Javascript integration by using `Splash`_ to render and execute Javascript.
Expected results A Scrapy middleware to integrate with `Splash`_
Required skills Scrapy
Mentor(s) Mikhail Korobov, Daniel Graña
==================== ===========
.. _Splash: https://github.com/scrapinghub/splash

26
GSoC-2014-Ideas.rst

@ -65,32 +65,6 @@ Mentor(s) Pablo Hoffman, Nicolas Ramirez
.. _SEP-019: https://github.com/scrapy/scrapy/blob/master/sep/sep-019.rst
Better generator support
------------------------
TODO:
This one should be better specified.
==================== ===========
Brief explanation Improve Scrapy API using generators
Expected results Scrapy should provide an easy way to build a single item from several pages, ...
Required skills Python, general understanding of async code, API design
Mentor(s) Mikhail Korobov, Rolando Espinoza
==================== ===========
There are areas where Scrapy usability and efficiency can be improved by using generators, for example:
* Integrate something like Rolando's https://github.com/darkrho/scrapy-inline-requests;
* ensure generators are not exhausted needlessly in various places;
* provide an easier alternative to spider_idle signal, something in line with https://github.com/scrapy/scrapy/issues/456
* ...
Reading list:
* http://www.python.org/dev/peps/pep-0342/
* http://www.tornadoweb.org/en/stable/gen.html
* http://twistedmatrix.com/documents/13.0.0/core/howto/defer.html
* https://twistedmatrix.com/trac/wiki/DeferredGenerator
Python 3 support
----------------

3
GSoC-2014.md

@ -1,4 +1,5 @@
Scrapy is applying for [Google Summer of Code 2014](http://www.google-melange.com/gsoc/homepage/google/gsoc2014)!
* [[GSoC 2014 Ideas]]
* [[GSoC 2014 Organization Application]]
* [[GSoC 2014 Organization Application]]
* [[GSoC 2014 Draft Ideas]]

14
GSoC-Draft-Ideas.rest

@ -1,14 +0,0 @@
Google Summer of Code - Draft Ideas
===================================
Improve javascript integration
------------------------------
==================== ===========
Brief explanation Improve Javascript integration by using `Splash`_ to render and execute Javascript.
Expected results A Scrapy middleware to integrate with `Splash`_
Required skills Scrapy
Mentor(s) Mikhail Korobov, Daniel Graña
==================== ===========
.. _Splash: https://github.com/scrapinghub/splash