RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Scrapy, a fast high-level web crawling & scraping framework for Python.
3,134 Commits 49 Branches 132 Tags 141 MiB
Python 99.8%
HTML 0.1%
Go to file
Pablo Hoffman 3d736e657f fixed typo in doc 2012-09-18 10:51:01 -03:00
.travis add precise to travis-ci build enviroments 2012-05-17 09:03:28 -03:00
artwork added artwork files properly now 2012-03-20 10:46:45 -03:00
bin return trial status as bin/runtests.sh exit value. #118 2012-04-20 16:40:55 -03:00
debian require w3lib 1.2 or greater 2012-05-15 17:11:51 -03:00
docs fixed typo in doc 2012-09-18 10:51:01 -03:00
extras simplified installation guide to only mention pip/easy_install mechanism, and provide hints for Windows users 2012-08-29 15:37:05 -03:00
scrapy make DBM the new default storage backend for HTTP cache middleware, simplified DBM storage backend code to avoid dealing with many spiders at once (not needed), and update httpcache stats names (hit -> hits, miss -> misses) 2012-09-17 10:11:07 -03:00
scrapyd simplified code for finished jobs times, updated docs for scrapyd 2012-09-12 11:10:49 +03:00
sep added sep directory with Scrapy Enhancement Proposal imported from old Trac site 2012-03-20 10:15:00 -03:00
.coveragerc Added rules to Makefile.buildbot for generating coverage reports 2010-12-15 11:13:45 -02:00
.gitignore Add .tox to .gitignore 2012-06-28 12:43:33 -07:00
.hgtags Added tag 0.10.3 for changeset 803efdb19e0b 2010-09-29 13:34:56 -03:00
.travis.yml add precise to travis-ci build enviroments 2012-05-17 09:03:28 -03:00
AUTHORS updated maintainer to scrapinghub 2012-05-02 03:25:35 -03:00
INSTALL mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
LICENSE mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
MANIFEST.in removed obsolete entries from MANIFEST.in 2012-05-17 01:09:48 -03:00
Makefile.buildbot moved coverage report script to extras/ 2010-12-15 12:08:59 -02:00
NEWS added NEWS file pointing to docs/news.rst 2012-04-28 23:32:51 -03:00
README.rst add travis-ci build status to README 2012-05-17 09:07:36 -03:00
setup.cfg simplified installation guide to only mention pip/easy_install mechanism, and provide hints for Windows users 2012-08-29 15:37:05 -03:00
setup.py require w3lib 1.2 or greater 2012-05-15 17:11:51 -03:00
tox.ini Add tox.ini for tox (http://tox.testrun.org/) 2012-06-28 12:43:12 -07:00

README.rst

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Scrapy

https://secure.travis-ci.org/scrapy/scrapy.png?branch=master

Overview

Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: http://scrapy.org

Requirements

  • Python 2.6 or up
  • Works on Linux, Windows, Mac OSX, BSD

Install

The quick way:

pip install scrapy

For more details see the install section in the documentation: http://doc.scrapy.org/en/latest/intro/install.html

Releases

You can download the latest stable and development releases from: http://scrapy.org/download/

Documentation

Documentation is available online at http://doc.scrapy.org/ and in the docs directory.

Community (blog, twitter, mail list, IRC)

See http://scrapy.org/community/

Contributing

See http://doc.scrapy.org/en/latest/contributing.html

Companies using Scrapy

See http://scrapy.org/companies/

Commercial Support

See http://scrapy.org/support/

</html>