mirror of https://github.com/scrapy/scrapy.git
Restored version
Rolando Espinoza La fuente
parent
c902c94e65
commit
60554a0d46
365
GSoC-2014.md
365
GSoC-2014.md
|
|
@ -1,364 +1,3 @@
|
|||
Scrapy is hoping to participate in Google Summer of Code in 2014. This page captures some ideas for student projects and provides information on how to get involved.
|
||||
# GSoC 2014
|
||||
|
||||
## Table of contents
|
||||
<!-- this table of contents is generated manually as there is no automatic way to do it (as far I have google it) -->
|
||||
|
||||
* [What is Scrapy?](#wiki-what-is-scrapy)
|
||||
* [Want to get involved?](#wiki-want-to-get-involved)
|
||||
* [Ideas for GSoC 2014](#wiki-ideas-for-gsoc-2014)
|
||||
* [Ideas around Scrapy](#wiki-ideas-around-scrapy)
|
||||
* [Better extension architecture and extensions repository](#wiki-better-extension-architecture-and-extensions-repository)
|
||||
* [Per-spider settings (SEP-19)](#wiki-per-spider-settings-sep-19)
|
||||
* [Better generator support](#wiki-better-generator-support)
|
||||
* [Python 3 support](#wiki-python-3-support)
|
||||
* [API from a Scrapy spider](#wiki-api-from-a-scrapy-spider)
|
||||
* [Better IPython integration](#wiki-better-ipython-integration)
|
||||
* [Profiling Scrapy](#wiki-profiling-scrapy)
|
||||
* [Improve JS integration](#wiki-improve-js-integration)
|
||||
* [Support for spiders other languages](#wiki-support-for-spiders-other-languages)
|
||||
* [Firefox Addon to build spiders](#wiki-firefox-addon-to-build-spiders)
|
||||
* [Multi-platform Scrapy GUI for running spiders](#wiki-multi-platform-scrapy-gui-for-running-spiders)
|
||||
* [Integration tests](#wiki-integration-tests)
|
||||
* [New HTTP1.1 download handler](#wiki-new-http11-download-handler)
|
||||
* [Refactor signal dispatcher](#wiki-refactor-signal-dispatcher)
|
||||
* [Around Scrapely](#wiki-around-scrapely)
|
||||
* [Replacing html regexp+python parsing with a c-based parser](#wiki-replacing-html-regexppython-parsing-with-a-c-based-parser)
|
||||
* [Better table support in scrapely](#wiki-better-table-support-in-scrapely)
|
||||
* [Support extracting fields in arbitrary order](#wiki-support-extracting-fields-in-arbitrary-order)
|
||||
* [Mentors](#wiki-mentors)
|
||||
|
||||
|
||||
# What is Scrapy?
|
||||
|
||||
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
|
||||
|
||||
Scrapy has a healthy and active community of developers. All development happens on the Scrapy GitHub project: https://github.com/scrapy/scrapy
|
||||
|
||||
# Want to get involved?
|
||||
|
||||
If you're interested in participating in GSoC 2014 as a student, you should join the Scrapy Developers mailing list: http://groups.google.com/group/scrapy-developers and post any questions, comments, etc. there.
|
||||
|
||||
You can also join the **#scrapy** IRC channel at **Freenode** to chat with other Scrapy users & developers.
|
||||
You can [join using the web browser](http://webchat.freenode.net/?channels=scrapy) or any IRC client.
|
||||
|
||||
# Ideas for GSoC 2014
|
||||
|
||||
From http://en.flossmanuals.net/GSoCMentoring/making-your-ideas-page/
|
||||
|
||||
> an Ideas Page should have the following:
|
||||
> * **Brief descriptions of projects that can be completed in about 12 weeks**.
|
||||
> * For each project, **a list of prerequisites**, description of programming skills needed and estimation of **difficulty level**.
|
||||
> * A list of potential mentors.
|
||||
>
|
||||
> In addition to a basic list, you might also consider providing links to relevant resources for mentors and students, particular FAQ entries, the timeline, etc. You might include a section on communication, giving specific advice on which mailing lists, channels and emails to use and how to use them.
|
||||
|
||||
Below is a list of potential projects for students. We've gathered these ideas from recent discussions on the mailing lists and among core developers.
|
||||
|
||||
This is for inspiration, you can apply with something completely different if you like. And if you do have another idea you'd like to propose, please consider sending it to http://groups.google.com/group/scrapy-developers for discussion.
|
||||
|
||||
## Ideas around Scrapy
|
||||
|
||||
These ideas are intended to make Scrapy easier to use, increase its performance (while we think it's already pretty good, there's still room for improvements!), and make interacting with it more straightforward.
|
||||
|
||||
### Better extension architecture and extensions repository
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Pablo Hoffman
|
||||
|
||||
Scrapy provides a lot of good hooks but makes it a bit complex
|
||||
to enable extensions and to list which extensions are enabled,
|
||||
partly because they are scattered around middlewares, pipelines, and bare extensions.
|
||||
|
||||
The idea here is to simplify Scrapy extensions from a user point of view.
|
||||
Doing things like
|
||||
|
||||
scrapy extension:add httpcache
|
||||
|
||||
and it would take care of configuring downloader middlewares, pipelines, and whatever is needed.
|
||||
And there would be a (searchable) repository of Scrapy extensions.
|
||||
|
||||
In other words, make it easier to plug new functionality to Scrapy projects
|
||||
which should encourage more writing more (and better) extensions
|
||||
|
||||
**_Pablo to write a new SEP document._**
|
||||
|
||||
### Per-spider settings (SEP-19)
|
||||
|
||||
**Brief explanation**: add support for overriding settings per-spider
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor** Pablo Hoffman or Nicolas Ramirez
|
||||
|
||||
This is a proposal to add support for overriding settings per-spiders in a consistent way.
|
||||
|
||||
In short, you will be able to override settings (on a per-spider basis) by implementing a class method in your spider:
|
||||
|
||||
def MySpider(BaseSpider):
|
||||
|
||||
@classmethod
|
||||
def custom_settings(cls):
|
||||
return {
|
||||
"DOWNLOAD_DELAY": 5.0,
|
||||
"RETRY_ENABLED": False,
|
||||
}
|
||||
|
||||
**What this solves**
|
||||
|
||||
* support true overridable per-spider settings, from both command-line usage and library mode
|
||||
* support for accessing settings from spiders (currently not supported without hacky code)
|
||||
* avoids mistakenly believing you can change settings after they have been populated (you can, but they won't have any effect)
|
||||
|
||||
More details and some implementation proposals can be found in https://github.com/scrapy/scrapy/blob/master/sep/sep-019.rst
|
||||
|
||||
### Better generator support
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Mikhail Korobov
|
||||
|
||||
Mikhail:
|
||||
> Integrating something like Rolando's https://github.com/darkrho/scrapy-inline-requests,
|
||||
> writing more tests to ensure generators are not exhausted needlessly,
|
||||
> and maybe adding a generator-based alternative to spider_idle signal.
|
||||
|
||||
### Python 3 support
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Mikhail Korobov
|
||||
|
||||
|
||||
Mikhail:
|
||||
> Python 3 porting project is quite hard because Twisted doesn't even install on Mac with Python 3
|
||||
> This project would require contributing not only Scrapy, but to Twisted as well.
|
||||
|
||||
Shane:
|
||||
> One good thing about this project is that it's easy to make progress and you can keep going.. I think it's OK if it doesn't go as far as windows or mac support.
|
||||
|
||||
### API from a Scrapy spider
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Shane Evans
|
||||
|
||||
Would take the same arguments as a Scrapy `Request` (url, callback,...)
|
||||
and would return items and requests
|
||||
|
||||
### Better IPython integration
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Mikhail Korobov
|
||||
|
||||
(Mikhail, Shane)
|
||||
|
||||
Mikhail:
|
||||
> develop IPython + Scrapy layer. It is possible to dislpay the HTML page inline in console, provide some interactive widgets and run Python code against the results (an old hacky demo of Scrapy+IPython is in attachements). IPython guys are going to release 2.0 version soon, and it should provide a standard protocol for such things.
|
||||
|
||||
### Profiling Scrapy
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Mikhail Korobov or Daniel Graña
|
||||
|
||||
Profile scrapy, benchmark it, find hot spots and fix them. Memory usage can be optimized as well. Maybe setup the benchmarking infrastructure.
|
||||
yup. Lots of room for perf improvement, in CPU and particularly memory. Better benchmarks would be great.
|
||||
|
||||
### Improve JS integration
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Pablo Hoffman or Mikhail Korobov
|
||||
|
||||
Shane:
|
||||
> maybe better scrapy-side support for splash or better scrapyjs integration
|
||||
|
||||
### Support for spiders other languages
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Shane Evans
|
||||
|
||||
Shane:
|
||||
> maybe something like hadoop streaming where we have a spider that forks a process and manages communication with it.
|
||||
|
||||
### Firefox Addon to build spiders
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Shane Evans
|
||||
|
||||
Illarion:
|
||||
> that helps to build spiders with complex logic visually,
|
||||
> like defining steps to get list of all items and steps to perform for each item just by selecting and clicking parts of the webpage.
|
||||
|
||||
### Multi-platform Scrapy GUI for running spiders
|
||||
|
||||
**Brief explanation**: Develop a multi-platform GUI interface for running Scrapy projects.
|
||||
|
||||
**Expected results**: A GUI interface that runs on Windows, Linux and Mac OSX. This interface is a companion to Scrapyd and must use (and possibly extend) the Scrapyd API. Basic features: schedule spider jobs (start, stop, pause), view/search items, view/filter/search logs, export items/logs.
|
||||
|
||||
**Required skills**: Multi-platform Python GUI development.
|
||||
|
||||
**Skill level**: Intermediate.
|
||||
|
||||
**Mentor**: (Rolando Espinoza)
|
||||
|
||||
### Integration tests
|
||||
|
||||
**Brief explanation**: Add integration tests for different networking scenarios
|
||||
|
||||
**Expected results**: Be able to tests from vertical to horizontal crawling against websites in same and different ips respecting throttling and handling timeouts, retries, dns failures. It must be simple to define new scenarios with predefined components (websites, proxies, routers, injected error rates)
|
||||
|
||||
**Required skills**: Python, Networking and Virtualization
|
||||
|
||||
**Skill level**: Medium
|
||||
|
||||
**Mentor**: Daniel Graña
|
||||
|
||||
### New HTTP1.1 download handler
|
||||
|
||||
**Brief explanation**: Replace current HTTP1.1 downloader handler with a in-house solution easily customizable to crawling needs.
|
||||
|
||||
**Expected results**: It must [degrade nicely to parse invalid responses](https://github.com/scrapy/scrapy/issues/345), filtering out the [offending headers](https://github.com/scrapy/scrapy/issues/210) as browsers does. It must be able to [avoid downloading responses bigger than a size limit](https://github.com/scrapy/scrapy/issues/336), it can be configured to [throttle bandwidth used per download](https://github.com/scrapy/scrapy/issues/157), and if there is enough time it can lay out the interface to [response streaming](https://github.com/scrapy/scrapy/issues/440)
|
||||
|
||||
**Required skills**: Python, Twisted and HTTP protocol
|
||||
|
||||
**Skill level**: Medium
|
||||
|
||||
**Mentor**: Daniel Graña
|
||||
|
||||
Current HTTP1.1 download handler depends on code shipped with Twisted that is not easily extensible by us, we ship twisted code under `scrapy.xlib.tx` to support running Scrapy in older twisted versions for distributions that doesn't ship uptodate Twisted packages. But this is an ongoing cat-mouse game, the http download handler is an essential component of a crawling framework and having no control over its release cycle leaves us with code that is hard to support.
|
||||
|
||||
The idea of this task is to depart from current Twisted code looking for a design that can cover current and future needs taking in count the goal is to deal with websites that doesn't follow standards to the letter.
|
||||
|
||||
### Refactor signal dispatcher
|
||||
|
||||
**Brief explanation**: Profile and look for alternatives to the backend of our signal dispatcher based on pydispatcher lib, Django moved out of pydispatcher by simplifying the api and improving its signal dispatching performance long time ago. [Scrapy issue #8](https://github.com/scrapy/scrapy/issues/8)
|
||||
|
||||
**Expected results**: Remove pydispatcher from libraries bundled with Scrapy and write a similar alternative based on Django signal dispatcher.
|
||||
|
||||
**Required skills**: Python
|
||||
|
||||
**Skill level**: Low
|
||||
|
||||
**Mentor**: Daniel Graña
|
||||
|
||||
## Around Scrapely
|
||||
|
||||
Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.
|
||||
|
||||
### Replacing html regexp+python parsing with a c-based parser
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Shane Evans
|
||||
|
||||
Shane:
|
||||
> e.g. gumbo parser, would need to research.
|
||||
> The goal would be both performance and parsing quality.
|
||||
|
||||
### Better table support in scrapely
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Mikhail Korobov
|
||||
|
||||
Shane:
|
||||
> (no need to annotate as much, or table-specific annotations)
|
||||
|
||||
### Support extracting fields in arbitrary order
|
||||
|
||||
**Brief explanation**: ---
|
||||
|
||||
**Expected results**: ---
|
||||
|
||||
**Required skills**: ---
|
||||
|
||||
**Skill level**: ---
|
||||
|
||||
**Mentor**: Shane Evans
|
||||
|
||||
Shane:
|
||||
> (interesting, bit of research, testing, etc.), or just where we have tables (or similar) and the field name is in one column with the value in another (easier version of the problem)
|
||||
|
||||
# Mentors
|
||||
|
||||
* Daniel Graña (@dangra)
|
||||
* Shane Evans (@shane42)
|
||||
* Pablo Hoffman (@pablohoffman)
|
||||
* Mikhail Korobov (@kmike)
|
||||
* Nicolas Ramirez (@nramirezuy) (potential)
|
||||
* Rolando Espinoza (@darkrho) (potential)
|
||||
See [GSoC 2014 Ideas](GSoC-2014-Ideas)
|
||||
Loading…
Reference in New Issue