RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,894 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

59 Commits

Author SHA1 Message Date
Paul Tremberth b05ae71852 Require at least lxml 2.3 2017-03-02 16:32:10 +01:00
Paul Tremberth 459ff0397d Only support Twisted prior to v17 2017-03-02 14:22:28 +01:00
Mikhail Korobov cbc250164f add service_identity to scrapy install_requires 2015-08-11 13:40:48 -03:00
Elias Dorneles 3d3633f3d2 favoring web scraping over screen scraping in the descriptions 2015-04-25 11:20:20 -03:00
Pablo Hoffman 5846d6154c emphasize web crawling over screen scraping on scrapy description. closes #586 2015-03-18 00:24:16 -03:00
Daniel Graña 0254f58763 Merge branch 'moderm-setuppy' 2014-08-08 18:09:43 -03:00
Daniel Graña 029c51acef There is a trove classifier for Scrapy framework! 
Added by https://bitbucket.org/pypa/pypi/issue/179
 2014-08-05 14:43:13 -03:00
Paul Tremberth 480cfa1994 Update w3lib requirement to 1.8.0 2014-08-04 16:01:28 +02:00
Daniel Graña fcd34b6561 set zip_safe=False 2014-08-01 16:26:08 -03:00
Daniel Graña 1fc4e59cf4 do not ship tests package 2014-08-01 16:23:37 -03:00
Daniel Graña a8f45dc6dd Modernize setup.py 2014-08-01 15:43:33 -03:00
Daniel Graña bfe02b019e Revert "limit Twisted support to pre-14.0.0 while #718 is fixed" 
This reverts commit 65f69e1609.
 2014-06-25 04:22:18 -03:00
Daniel Graña 65f69e1609 limit Twisted support to pre-14.0.0 while #718 is fixed 2014-06-02 13:27:20 -03:00
alexanderlukanin13 ab6496647c Added "six>=1.5.2" to requirements 2014-01-15 13:05:00 +06:00
Daniel Graña cfe588103c include_package_data is required to build wheels from published sources 2013-11-28 16:25:57 -02:00
Daniel Graña 2df8156431 Drop Python 2.6 support 2013-10-29 13:44:00 -02:00
alexanderlukanin13 192c323a0c wrong variable name 2013-10-18 11:30:48 +06:00
Daniel Graña 757e262267 update requirement for cssselect >= 0.9 2013-10-17 09:33:15 -02:00
Capi Etheriel bc17e9d412 Adds HtmlCSSSelector and XmlCSSSelector classes, cssselect as optional dependency. 
Ported .get() from _Element and .text_content() from HTMLMixin

Add CSS selectors to scrapy shell

Documenting CSS Selectors: Constructing selectors

Documenting CSS Selectors: Using Selectors

Make CSS Selectors a default feature.

Adds XPath powers to CSS Selectors and some syntactic sugar.

Removes methods copied over from lxml.html.HtmlMixin.

Updating docs to use new CSS Selector super powers.

Documenting CSS Selectors: Regular Expressions

Moving section after Nesting section, since it mentions it.

Documenting CSS Selectors: Nesting Selectors

Fix XPath specificity in lxml.selector.CSSSelectorMixin.text

Cleaning up unused stuff from cssel.py

Changing the behavior of lxml.selector.CSSSelectorMixin.text.

Concatenating all of the descendant text nodes is more useful
than returning it in pieces (there's xpath() if you need that).

Documenting CSS Selectors: CSS Selector objects

Documenting CSS Selectors: CSSSelectorList objects

Documenting CSS Selectors: HtmlCSSSelector objects

Documenting CSS Selectors: XmlCSSSelector objects

Fixing some documentations typos and errors

Enforcing the 80-char width lines

Tidying up CSS selectors and CSSSelectorMixin objects

Adding some missing references in documentation.

Fixing lxml.selector.CSSSelectorList.text
 2013-10-10 18:23:15 -02:00
Pablo Hoffman a9c3519897 updated required twisted version to 10.0 2013-10-01 14:07:38 -03:00
Pablo Hoffman d02da2f31f ported code to use queuelib 2013-04-23 17:48:09 -03:00
Daniel Graña 910effd145 get scrapy version from package data 2013-02-06 11:44:26 -02:00
Daniel Graña 5db45b3825 remove scrapyd, it was migrated to its own repository 2013-02-06 05:24:07 +00:00
Pablo Hoffman b9a96147ed setup.py: use README.rst to populate long_description 2012-11-25 22:22:33 -02:00
Daniel Graña d74a067227 require w3lib 1.2 or greater 2012-05-15 17:11:51 -03:00
Pablo Hoffman 4f28ffcb2c removed no longer needed dependency on simplejson 2012-04-10 16:01:36 -03:00
Pablo Hoffman 6e8edbd72e switched default selectors backend to lxml 2012-04-10 15:52:14 -03:00
Pablo Hoffman b6ae266546 Removed (very old and possibly broken) backwards compatibility support for Twisted 2.5 2012-03-15 00:28:24 -03:00
Pablo Hoffman e521da2e2f Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25 2012-03-01 08:18:12 -02:00
Pablo Hoffman de3a3b68dc bumped required w3lib version to 1.1, after refactoring encoding detection to use the new w3lib.encoding module 2012-02-29 07:44:22 -02:00
Pablo Hoffman b6220b8e95 use git describe for building version from git, and removed support for building version from hg 2012-01-02 13:05:26 -02:00
Pablo Hoffman 1e43afeaea added support for generating version from git revision, and use it in extras/makedeb.py 2011-09-09 03:03:46 -03:00
Pablo Hoffman 804c0279ec setup.py: only add lxml requirement if libxml2 is not available 2011-07-13 13:04:42 -03:00
Pablo Hoffman 7f97259ba7 added w3lib to requirements, in installation guide 2011-05-01 11:14:57 -03:00
Pablo Hoffman b10f4fae35 Moved several functions from scrapy.utils.{http,markup,multipart,response,url} (and their tests) to a new library called 'w3lib'. Scrapy will now depend on w3lib. 2011-04-18 22:37:19 -03:00
Pablo Hoffman 97a790d659 improved mechanism for tagging version with hg revision 2010-12-13 10:21:39 -02:00
Pablo Hoffman 7c712eeda1 Removed scrapy.xlib.simplejson module. Scrapy now requires simplejson if running on Python 2.5. Closes #289 2010-11-16 03:11:12 -02:00
Pablo Hoffman df63577243 added lxml to setup.py install_requires, if using setuptools. refs #147 2010-10-25 14:57:50 -02:00
Pablo Hoffman 9206806770 setup.py: added support for generating version from hg revision 2010-09-27 08:19:32 -03:00
Pablo Hoffman b16eff227b Bumped version to 0.10 2010-09-10 01:48:45 -03:00
Pablo Hoffman 37e9c5d78e Added new Scrapy service with support for: 
* multiple projects
* uploading scrapy projects as Python eggs
* scheduling spiders using a JSON API

Documentation is added along with the code.

Closes #218.

--HG--
rename : debian/scrapy-service.default => debian/scrapyd.default
rename : debian/scrapy-service.dirs => debian/scrapyd.dirs
rename : debian/scrapy-service.install => debian/scrapyd.install
rename : debian/scrapy-service.lintian-overrides => debian/scrapyd.lintian-overrides
rename : debian/scrapy-service.postinst => debian/scrapyd.postinst
rename : debian/scrapy-service.postrm => debian/scrapyd.postrm
rename : debian/scrapy-service.upstart => debian/scrapyd.upstart
rename : extras/scrapy.tac => extras/scrapyd.tac
 2010-09-03 15:54:42 -03:00
Pablo Hoffman e7b3247a18 Updated some missing references to scrapy-ws script 2010-08-27 01:05:59 -03:00
Pablo Hoffman 8e83f527b3 Removed scrapy-sqs script, as it has been superseded by the new scrapy 'queue' command 2010-08-23 22:04:49 -03:00
Pablo Hoffman 34554da201 Deprecated scrapy-ctl.py command in favour of simpler "scrapy" command. Closes #199. Also updated documenation accordingly and added convenient scrapy.bat script for running from Windows. 
--HG--
rename : debian/scrapy-ctl.1 => debian/scrapy.1
rename : docs/topics/scrapy-ctl.rst => docs/topics/cmdline.rst
 2010-08-18 19:48:32 -03:00
Pablo Hoffman d17695ee4b Added "scrapy" command with project settings auto-discovery. Refs #199 2010-08-17 00:59:24 -03:00
Pablo Hoffman 2349d241e0 removed custom Makefile and version based on mercurial revision 2010-07-24 17:02:08 -03:00
Pablo Hoffman 2d3d5b6aba use mercurial revision to construct version, when building a non-final version 2010-06-14 08:59:20 -03:00
Pablo Hoffman 68e8e6ac11 removed unused code 2010-06-14 08:28:48 -03:00
Pablo Hoffman 09182efaff added scrapy-sqs.py to deployed scripts 2010-06-13 19:17:17 -03:00
Pablo Hoffman ea8b5ddfd5 debian package: fix dh_auto_build confusing with Makefile, added scrapy-ws.py to deployed scripts 2010-06-11 12:48:35 -03:00