RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

add mentor to asyncio idea, fix mentor links

Steven Almeroth 2015-02-12 09:26:30 -06:00
parent a93e9175b4
commit 2281f53647
1 changed files with 15 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
GSoC-2015-Ideas.rest

@ -102,7 +102,7 @@ Scrapy integration tests
------------------------
==================== ======================
Brief explanation Add integration tests for different networking scenarios
Brief explanation Add integration tests for different networking scenarios
Expected results Be able to tests from vertical to horizontal crawling against websites in same and different ips respecting throttling and handling timeouts, retries, dns failures. It must be simple to define new scenarios with predefined components (websites, proxies, routers, injected error rates)
Required skills Python, Networking and Virtualization
Difficulty level Intermediate
@ -113,7 +113,7 @@ New HTTP/1.1 download handler
-----------------------------
==================== ======================
Brief explanation Replace current HTTP1.1 downloader handler with a in-house solution easily customizable to crawling needs. Current HTTP1.1 download handler depends on code shipped with Twisted that is not easily extensible by us, we ship twisted code under `scrapy.xlib.tx` to support running Scrapy in older twisted versions for distributions that doesn't ship uptodate Twisted packages. But this is an ongoing cat-mouse game, the http download handler is an essential component of a crawling framework and having no control over its release cycle leaves us with code that is hard to support. The idea of this task is to depart from current Twisted code looking for a design that can cover current and future needs taking in count the goal is to deal with websites that don't follow standards to the letter.
Brief explanation Replace current HTTP1.1 downloader handler with a in-house solution easily customizable to crawling needs. Current HTTP1.1 download handler depends on code shipped with Twisted that is not easily extensible by us, we ship twisted code under `scrapy.xlib.tx` to support running Scrapy in older twisted versions for distributions that doesn't ship uptodate Twisted packages. But this is an ongoing cat-mouse game, the http download handler is an essential component of a crawling framework and having no control over its release cycle leaves us with code that is hard to support. The idea of this task is to depart from current Twisted code looking for a design that can cover current and future needs taking in count the goal is to deal with websites that don't follow standards to the letter.
Expected results A HTTP parser that degrades nicely to `parse invalid responses`_, filtering out the `offending headers`_ and `cookies`_ as browsers does. It must be able to avoid downloading responses bigger than a `size limit`_, it can be configured to `throttle bandwidth`_ used per download, and if there is enough time it can lay out the interface to `response streaming`_ and support features such as `HTTP pipelining`_
Required skills Python, Twisted and HTTP protocol
@ -150,34 +150,37 @@ Brief explanation The `asyncio`_ library provides infrastructure for writing
Expected results A simple proof of concept of an asyncio based Scrapy
Required skills Python
Difficulty level Advanced
Mentor(s) |juan|_
Mentor(s) |juan|_, |steven|_
==================== ======================
.. _asyncio: https://docs.python.org/3/library/asyncio.html
.. |daniel| replace:: Daniel Graña
.. _daniel: GSoC-2015-Mentors#wiki-daniel
.. _daniel: GSoC-2015-Mentors#daniel-gra%C3%B1a
.. |mikhail| replace:: Mikhail Korobov
.. _mikhail: GSoC-2015-Mentors#wiki-mikhail
.. _mikhail: https://github.com/scrapy/scrapy/wiki/GSoC-2015-Mentors#mikhail-korobov
.. |nicolas| replace:: Nicolas Ramirez
.. _nicolas: GSoC-2015-Mentors#wiki-nicolas
.. _nicolas: https://github.com/scrapy/scrapy/wiki/GSoC-2015-Mentors#nicolas-ramirez
.. |pablo| replace:: Pablo Hoffman
.. _pablo: GSoC-2015-Mentors#wiki-pablo
.. _pablo: https://github.com/scrapy/scrapy/wiki/GSoC-2015-Mentors#pablo-hoffman
.. |paul| replace:: Paul Tremberth
.. _paul: GSoC-2015-Mentors#wiki-paul
.. _paul: https://github.com/scrapy/scrapy/wiki/GSoC-2015-Mentors#paul-tremberth
.. |rolando| replace:: Rolando Espinoza
.. _rolando: GSoC-2015-Mentors#wiki-rolando
.. _rolando: https://github.com/scrapy/scrapy/wiki/GSoC-2015-Mentors#rolando-espinoza
.. |shane| replace:: Shane Evans
.. _shane: GSoC-2015-Mentors#wiki-shane
.. _shane: https://github.com/scrapy/scrapy/wiki/GSoC-2015-Mentors#shane-evans
.. |julia| replace:: Julia Medina
.. _julia: GSoC-2015-Mentors#wiki-julia
.. _julia: https://github.com/scrapy/scrapy/wiki/GSoC-2015-Mentors#julia-medina
.. |juan| replace:: Juan Riaza
.. _juan: GSoC-2015-Mentors#wiki-juan
.. _juan: https://github.com/scrapy/scrapy/wiki/GSoC-2015-Mentors#juan-riaza
.. |steven| replace:: Steven Almeroth
.. _steven: https://github.com/scrapy/scrapy/wiki/GSoC-2015-Mentors#steven-almeroth