RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,188 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

4188 Commits

Author SHA1 Message Date
Paul Tremberth ed1f376d2b Extract links from XHTML documents with MIME-Type "application/xml" 
"application/xhtml+xml" is already interpreted as HTML
and link extractor is fine with it to extract links.

Only for XML documents can the namespaces in tags be an issue.

Fixes #780

Do both tag and attribute tests in _iter_links() method
 2014-07-11 17:15:16 -03:00
Pablo Hoffman 91a110622b Merge pull request #793 from roysc/patch-1 
Fix typo in commands.rst
 2014-07-10 13:27:10 -03:00
Roy Crihfield 743e1e2034 Fix typo in commands.rst 2014-07-10 11:04:48 -05:00
Daniel Graña e22daaf411 better testcase for settings.overrides.setdefault 2014-07-09 01:42:10 -03:00
ivan 5ec430baa7 Using CRLF as line marker according to http 1.1 definition 2014-07-09 01:42:04 -03:00
Daniel Graña 2cba7b5dee Add 0.24.2 release notes 2014-07-08 05:53:08 -03:00
Daniel Graña 37af19296e Bump version: 0.24.1 → 0.24.2 2014-07-08 05:51:21 -03:00
Daniel Graña e5e81335e7 Use a mutable mapping to proxy deprecated settings.overrides and settings.defaults attribute 2014-07-08 05:45:23 -03:00
Daniel Graña 3cd6146e19 there is not support for python3 yet 2014-07-01 17:12:11 +00:00
Daniel Graña fa5d76b2df Update python compatible version set to debian packages 2014-07-01 13:50:46 -03:00
Mikhail Korobov c6a9e20473 DOC fix formatting in release notes 2014-06-28 13:29:56 +06:00
Daniel Graña 620f712e20 Add 0.24.1 release notes 2014-06-27 12:02:41 -03:00
Daniel Graña 55e6d06c8d Bump version: 0.24.0 → 0.24.1 2014-06-27 12:00:10 -03:00
Daniel Graña 8e3f20a4a9 Fix deprecated CrawlerSettings and increase backwards compatibility with .defaults attribute 2014-06-27 11:58:56 -03:00
Daniel Graña af23770573 set 0.24.0 release date 2014-06-26 11:45:40 -03:00
Daniel Graña f0facb9793 Bump version: 0.23.0 → 0.24.0 2014-06-26 10:30:37 -03:00
Daniel Graña ee33efac39 Add 0.24.0 release notes 2014-06-26 10:29:47 -03:00
Daniel Graña eb97d257f0 Merge pull request #764 from dangra/763-bugfix 
no need to keep links as instance attribute
 2014-06-25 16:29:15 -03:00
Daniel Graña 53057728fb No need to keep extracted links as instance attribute. fixes #763 2014-06-25 16:11:35 -03:00
Daniel Graña ccec728c3a Merge pull request #762 from kmike/688-spider-templates 
add utf8 encoding header to spider templates
 2014-06-25 15:56:33 -03:00
Mikhail Korobov 6fb7093154 add utf8 encoding header to spider templates 2014-06-26 00:30:27 +06:00
Mikhail Korobov df4ba41c82 Merge pull request #688 from stillzhl/master 
add encoding utf-8 to the first line
 2014-06-26 00:26:12 +06:00
Daniel Graña 436c1c8980 address latest comments 2014-06-25 15:12:58 -03:00
Daniel Graña 2ad8db6ae6 Merge pull request #761 from dangra/lxmlextractor 
Promote LxmlLinkExtractor as LxmlExtractor
 2014-06-25 15:07:02 -03:00
Daniel Graña a9ecef5662 promote LxmlLinkExtractor as default in docs 2014-06-25 14:34:30 -03:00
Mikhail Korobov ef81127b99 Merge pull request #760 from dangra/546-remaining 
recognize jl extension as jsonlines exporter and update docs
 2014-06-25 23:31:48 +06:00
Daniel Graña 5b2faf61c3 recognize jl extension as jsonlines exporter and update docs 2014-06-25 13:55:15 -03:00
Daniel Graña f98bf2c521 Merge pull request #737 from Curita/settings-cleanup 
Settings api cleanup
 2014-06-25 13:01:50 -03:00
Daniel Graña f224ac1328 Restore compatibility with Settings.overrides while still deprecating it 2014-06-25 13:01:12 -03:00
Daniel Graña f72b1597b4 Merge pull request #718 from dangra/twisted-14.0.0 
[WIP] does it work with Twisted 14.0.0?
 2014-06-25 09:50:31 -03:00
Daniel Graña d48158427a hold a reference to backwards compatible _contextFactory 2014-06-25 04:22:18 -03:00
Daniel Graña bfe02b019e Revert "limit Twisted support to pre-14.0.0 while #718 is fixed" 
This reverts commit 65f69e1609.
 2014-06-25 04:22:18 -03:00
Daniel Graña 559af943dc Add bumpversion config 2014-06-25 03:19:04 -03:00
Daniel Graña 1b32ece918 Merge branch 'settings-cleanup' of https://github.com/Curita/scrapy into Curita-settings-cleanup 2014-06-25 02:55:20 -03:00
Daniel Graña 4300334a23 Upload sdist and wheel packages to pypi using travis-ci deploys 2014-06-25 02:53:12 -03:00
Daniel Graña 499438ec44 indent parsed-literal as part of ordered list 2014-06-24 10:32:56 -03:00
Pablo Hoffman ee5c3209af doc: use |version| substitution in ubuntu packages 2014-06-24 09:52:51 -03:00
Paul Tremberth 90e69141d3 Add doc on LxmlLinkExtractor class 2014-06-23 22:35:56 +02:00
Paul Tremberth d34a57a10e Add LxmlLinkExtractor class similar to SgmlLinkExtractor (#528) 2014-06-23 22:35:56 +02:00
Daniel Graña 05ffca2781 Merge pull request #626 from dangra/500-leveldb-cache 
[WIP] Add a LevelDB cache backend
 2014-06-23 15:30:13 -03:00
Daniel Graña a90fd81c97 add leveldb cache backend docs 2014-06-23 14:37:29 -03:00
Daniel Graña 9d57ecfc35 Merge pull request #752 from tpeng/slightly-improve-contract-tests 
print spider name even it has no contract tests when -v is specified
 2014-06-18 03:15:21 -03:00
Daniel Graña 7449b25bed Merge pull request #745 from crlane/cl/offsite-middleware-bugfix 
Cl/offsite middleware bugfix
 2014-06-18 03:01:21 -03:00
tpeng 5fb22ba4e7 print spider name even it has no contract tests when -v is specified 2014-06-17 13:39:10 +02:00
Pablo Hoffman 7fd8a1a387 minor change to request_scheduled signal doc 2014-06-14 20:51:16 -03:00
Pablo Hoffman 313ff26a79 Merge pull request #746 from Digenis/doc 
Document signal "request_scheduled"
 2014-06-14 20:49:43 -03:00
Nikolaos-Digenis Karagiannis 5f79940295 Document signal "request_scheduled" 2014-06-14 11:08:18 +03:00
Cameron Lane a069d36652 [#744] Test for allowed domains including NoneTypes 2014-06-12 11:43:03 -04:00
Cameron Lane eefa878f98 [#744] Ensure domain is not None before building regex 2014-06-12 11:42:14 -04:00
Pablo Hoffman ad11fc01cb crate release notes for 0.24 and #699 to it 2014-06-10 18:46:53 -03:00