RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Scrapy, a fast high-level web crawling & scraping framework for Python.
3,576 Commits 49 Branches 132 Tags 141 MiB
Python 99.8%
HTML 0.1%
Go to file
Mikhail Korobov 7ffc786fb6 mock is required for running tests since 45ff6ec28a 
This fixes tox test running. Mock is available at Travis, but there is no harm in listing it explicitly.
 		2013-09-18 00:02:23 +06:00
.travis mock is required for running tests since 45ff6ec28a 2013-09-18 00:02:23 +06:00
artwork added artwork files properly now 2012-03-20 10:46:45 -03:00
bin remove scrapyd, it was migrated to its own repository 2013-02-06 05:24:07 +00:00
debian update copyright notes 2013-05-16 15:05:52 -03:00
docs Fixed directory location for dmoz_spider.py file 2013-09-15 21:55:52 -07:00
extras get scrapy version from package data 2013-02-06 11:44:26 -02:00
scrapy forms: do not submit reset inputs 2013-09-07 21:53:27 -03:00
sep sep-019: fixed typo 2013-03-12 18:44:09 -03:00
.coveragerc Added rules to Makefile.buildbot for generating coverage reports 2010-12-15 11:13:45 -02:00
.gitignore Instead of extending from HttpCachePolicy, following the same approach used for storage selection 2012-12-28 16:11:47 +01:00
.travis.yml no need to test latest versions of dependencies on python 2.6 2013-08-27 17:38:28 -03:00
AUTHORS added Nicolas Ramirez to AUTHORS 2013-03-14 12:44:39 -03:00
CONTRIBUTING.md renamed CONTRIBUTING to CONTRIBUTING.md so that links are rendered as links in github 2012-09-19 13:58:58 -03:00
INSTALL fix link to online installation instructions 2012-10-02 12:26:14 +01:00
LICENSE mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
MANIFEST.in get scrapy version from package data 2013-02-06 11:44:26 -02:00
Makefile.buildbot Fix permission and set umask before generating sdist tarball 2013-09-03 23:22:17 -03:00
NEWS added NEWS file pointing to docs/news.rst 2012-04-28 23:32:51 -03:00
README.rst add travis-ci build status to README 2012-05-17 09:07:36 -03:00
requirements.txt fixed scrapy.utils.gz.gunzip() broken after changes from Python 2.7.3 to 2.7.4 2013-08-09 18:20:04 -03:00
setup.cfg simplified installation guide to only mention pip/easy_install mechanism, and provide hints for Windows users 2012-08-29 15:37:05 -03:00
setup.py ported code to use queuelib 2013-04-23 17:48:09 -03:00
tox.ini tox.ini: disable sitepackages on windows, as a compiler is often not available 2013-08-12 18:59:41 -03:00

README.rst

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Scrapy

https://secure.travis-ci.org/scrapy/scrapy.png?branch=master

Overview

Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: http://scrapy.org

Requirements

  • Python 2.6 or up
  • Works on Linux, Windows, Mac OSX, BSD

Install

The quick way:

pip install scrapy

For more details see the install section in the documentation: http://doc.scrapy.org/en/latest/intro/install.html

Releases

You can download the latest stable and development releases from: http://scrapy.org/download/

Documentation

Documentation is available online at http://doc.scrapy.org/ and in the docs directory.

Community (blog, twitter, mail list, IRC)

See http://scrapy.org/community/

Contributing

See http://doc.scrapy.org/en/latest/contributing.html

Companies using Scrapy

See http://scrapy.org/companies/

Commercial Support

See http://scrapy.org/support/

</html>