From 7f78f6f6a6830d1a2fc3b0654e93d12c2f657416 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Thu, 6 Feb 2014 13:14:19 -0800 Subject: [PATCH] Updated GSoC 2014 (markdown) --- GSoC-2014.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/GSoC-2014.md b/GSoC-2014.md index a291f9a..e4f96b4 100644 --- a/GSoC-2014.md +++ b/GSoC-2014.md @@ -1,5 +1,11 @@ Scrapy is hoping to participate in Google Summer of Code in 2014. This page captures some ideas for student projects and provides information on how to get involved. +# What is Scrapy? + +Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. + +Scrapy has a healthy and active community of developers. All development happens on the Scrapy GitHub project: https://github.com/scrapy/scrapy + # Want to get involved? If you're interested in participating in GSoC 2014 as a student, you should join the Scrapy Developers mailing list: http://groups.google.com/group/scrapy-developers and post any questions, comments, etc. there. @@ -9,13 +15,11 @@ You can [join using the web browser](http://webchat.freenode.net/?channels=scrap # Ideas for GSoC 2014 -Below is a list of potential projects for students. We've gathered these from recent discussions on the mailing lists and among core developers. +Below is a list of potential projects for students. We've gathered these ideas from recent discussions on the mailing lists and among core developers. -This is for inspiration, you can apply with something completely different if you like. +This is for inspiration, you can apply with something completely different if you like. And if you do have another idea you'd like to propose, please consider sending it to http://groups.google.com/group/scrapy-developers for discussion. -And if you have another idea you'd like to propose, please consider sending it to http://groups.google.com/group/scrapy-developers for discussion. - -## Around Scrapy itself +## Ideas around Scrapy These ideas are intended to make Scrapy easier to use, increase its performance (while we think it's already pretty good, there's still room for improvements!), and make interacting with it more straightforward. @@ -25,7 +29,8 @@ Scrapy provides a lot of good hooks but makes it a bit complex to enable extensions and to list which extensions are enabled, partly because they are scattered around middlewares, pipelines, and bare extensions. -The idea here is to simplify Scrapy extensions from a user point of view. Doing things like +The idea here is to simplify Scrapy extensions from a user point of view. +Doing things like scrapy extension:add httpcache @@ -41,7 +46,7 @@ which should encourage more writing more (and better) extensions This is a proposal to add support for overriding settings per-spiders in a consistent way. -In short, you will be able to overwrite settings (on a per-spider basis) by implementing a class method in your spider: +In short, you will be able to override settings (on a per-spider basis) by implementing a class method in your spider: def MySpider(BaseSpider): @@ -54,7 +59,7 @@ In short, you will be able to overwrite settings (on a per-spider basis) by impl **What this solves** -* support true overridable per-spider setting, from both command-line usage and library mode +* support true overridable per-spider settings, from both command-line usage and library mode * support for accessing settings from spiders (currently not supported without hacky code) * avoids mistakenly believing you can change settings after they have been populated (you can, but they won't have any effect) @@ -118,6 +123,8 @@ The features could be: ## Around Scrapely +Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages. + ### Replacing html regexp+python parsing with a c-based parser Shane: > e.g. gumbo parser, would need to research.