RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,878 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

3878 Commits

Author SHA1 Message Date
Benoit Blanchon 7e4d62778e BaseSgmlLinkExtractor: Fixed unknown_endtag() so that it only set current_link=None when the end tag match the opening tag 2014-02-05 20:03:52 -02:00
Paul Tremberth 76c7e200aa Fix tests for Travis-CI build 2014-02-05 20:01:53 -02:00
Daniel Graña 5f87b17047 replace unencodeable codepoints with html entities. fixes #562 and #285 2014-02-05 20:01:13 -02:00
Paul Tremberth d0ee545a8f RegexLinkExtractor: encode URL unicode value when creating Links 2014-02-05 20:01:00 -02:00
Rolando Espinoza 8da65deda1 Updated the tutorial crawl output with latest output. 2014-02-05 20:00:33 -02:00
Rolando Espinoza 875b9abe8a Updated shell docs with the crawler reference and fixed the actual shell output. 
Also updated the shell example with a reproducible code example.
 2014-02-05 20:00:28 -02:00
Rolando Espinoza f89efaf667 PEP8 minor edits. 2014-02-05 20:00:23 -02:00
Rolando Espinoza 5349cecd0e Expose current crawler in the scrapy shell. 2014-02-05 20:00:18 -02:00
Rolando Espinoza 387f414894 Unused re import and PEP8 minor edits. 2014-02-05 19:59:37 -02:00
Rolando Espinoza 06325467d8 Ignore None's values when using the ItemLoader. 2014-02-05 19:59:22 -02:00
Rolando Espinoza cde9a8c68d DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm. 2014-01-30 22:54:26 +06:00
Daniel Graña fb5c9c552f show ubuntu setup instructions as literal code 2014-01-20 16:23:12 -02:00
Daniel Graña 70fb1058ff Update Ubuntu installation instructions 2014-01-20 15:32:26 -02:00
Daniel Graña 6f70b6a196 Merge pull request #550 from stray-leone/patch-1 
modify the version of scrapy ubuntu package
 2014-01-20 04:08:39 -08:00
stray-leone 725900df44 modify the version of scrapy ubuntu package 
latest version is 0.22.
with scrapy-0.18, tutorial project provides error
relative issue : https://github.com/scrapy/scrapy/issues/511
 2014-01-20 12:32:09 +09:00
Daniel Graña af0219a561 fix 0.22.0 release date 2014-01-17 17:55:34 -02:00
Mikhail Korobov b7f58f47a8 fix typos in news.rst and remove (not released yet) header 2014-01-18 01:44:21 +06:00
Daniel Graña 8fee933c15 bumped version to 0.22.0 2014-01-17 15:49:39 -02:00
Mikhail Korobov 7b7a1d8dfd Make Filesystem storage backend default again. See GH-500. 2014-01-17 04:32:08 +06:00
Daniel Graña f0851e41ec Merge pull request #478 from redapple/offsitetests 
Add tests for OffsiteMiddleware() + use re.escape() in domains regexp
 2014-01-16 14:02:26 -08:00
Daniel Graña 151f9478c1 Merge pull request #538 from kmike/ajaxcrawlable-rename 
[MRG] Rename AjaxCrawlableMiddleware to AjaxCrawlMiddleware
 2014-01-16 10:33:46 -08:00
Mikhail Korobov b03fe04999 Rename AjaxCrawlableMiddleware to AjaxCrawlMiddleware 2014-01-16 23:09:37 +06:00
Pablo Hoffman ed6fd4933f Merge pull request #524 from hobsonlane/master 
documentation code example corrections per pablohoffman
 2014-01-16 06:44:51 -08:00
Pablo Hoffman 71ada5476e Merge pull request #472 from redapple/exslt 
Register EXSLT namespaces by default (resolves #470)
 2014-01-16 06:32:05 -08:00
Daniel Graña b9bb9bed6b Merge pull request #343 from kmike/ajax-crawlable 
[MRG] AjaxCrawlableMiddleware
 2014-01-16 05:07:39 -08:00
Daniel Graña e2a5310f92 Merge pull request #537 from dangra/warn-xpathselector-subclass 
warn XPathSelector deprecation on subclassing and direct instance
 2014-01-16 04:14:27 -08:00
Daniel Graña 5a175ad287 Merge pull request #519 from dangra/warn-once 
Warn BaseSpider deprecation only once
 2014-01-16 04:07:21 -08:00
Daniel Graña b3be6e210d warn XPathSelector deprecation on subclassing and direct instance 2014-01-16 09:53:59 -02:00
Daniel Graña 3e42646ce1 do not test multiple instantation warnings 2014-01-16 09:49:51 -02:00
Hobson Lane 85a80d0752 remove "for brevity's sake" line and correct "Torrent item" 
Torrent item -> TorrentItem class
 2014-01-15 17:29:23 -08:00
Mikhail Korobov 03dab0117d Merge pull request #536 from manfre/patch-1 
Fix comment typo
 2014-01-15 16:39:33 -08:00
Michael Manfre 4a2a45b495 Fix comment typo 2014-01-15 12:35:34 -05:00
Paul Tremberth 827c0cf51f Rename "regexp" prefix to "re" 2014-01-15 15:00:25 +01:00
Mikhail Korobov c92f52ce2c Merge pull request #525 from alexanderlukanin13/urllib_test_coverage 
Improved test coverage
 2014-01-15 04:05:37 -08:00
Mikhail Korobov af16fa326f Merge pull request #533 from chekunkov/make_dupefilter_easily_subclassable 
Make dupefilter easily subclassable
 2014-01-15 03:56:45 -08:00
Paul Tremberth 88c8a523a7 Add warning in docs on performance when using EXSLT regexp functions 2014-01-15 12:52:10 +01:00
Alexander Chekunkov 5411695905 removed request_fingerprint method from BaseDupeFilter, removed unnecessary docstring 2014-01-15 13:44:46 +02:00
Paul Tremberth a3eba68aca Drop EXSLT strings and math extensions 2014-01-15 12:28:25 +01:00
Alexander Chekunkov c2421a4b3f Added request_fingerprint method to dupefilter classes so they could be easily subclassed without need to override entire request_seen method. 2014-01-15 13:02:05 +02:00
alexanderlukanin13 25759bdf5c Removed deploy test (as deploy command is deprecated and moved to scrapyd) 2014-01-15 13:09:36 +06:00
alexanderlukanin13 ab6496647c Added "six>=1.5.2" to requirements 2014-01-15 13:05:00 +06:00
Pablo Hoffman ea2f897b81 Merge pull request #502 from scrapy/doc-fixes 
DOWNLOAD_DELAY docs clarification
 2014-01-14 21:07:42 -08:00
Hobson Lane a3db95985b another import name correction by pablo 2014-01-14 21:04:15 -08:00
Pablo Hoffman 3d308d7732 Merge pull request #517 from hoprocker/patch-1 
Clarify return value from extract_links
 2014-01-14 21:01:32 -08:00
Pablo Hoffman 622ffd84d0 Merge pull request #523 from dangra/warn-xpathitemloader-subclass 
warn on XPathItemLoader subclassing
 2014-01-14 20:58:07 -08:00
Pablo Hoffman b377d4c28c Merge pull request #531 from kmike/fix-memdebug 
Fix logging of stats collected by MemoryDebugger extension.
 2014-01-14 20:42:19 -08:00
Mikhail Korobov 26538b01b5 Fix logging of stats collected by MemoryDebugger extension. 
Stats are printed on spider_closed event;
engine_stopped signal is called after spider_closed signal,
so stats for MemoryDebugger extension were not printed to user.
 2014-01-15 04:05:09 +06:00
Daniel Graña 553c70a9dc Merge pull request #530 from kmike/memusage-osx 
Fix MemoryUsage extension for OS X
 2014-01-14 13:56:02 -08:00
Mikhail Korobov a83c6f545c Fix MemoryUsage extension for OS X. Fix GH-529. 2014-01-15 03:24:48 +06:00
Paul Tremberth 2cc26e6f56 Fix typo error 2014-01-14 13:09:18 +01:00