RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,368 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

907 Commits

Author SHA1 Message Date
Julia Medina ed033854e2 setmodule helper method on Settings class 2014-06-10 11:26:50 -03:00
Julia Medina bdca06240c Fix settings repr on the logs of the shell and tutorial docs topics 2014-06-10 11:26:50 -03:00
Julia Medina 8a3a6236d9 Settings topic updated 2014-06-10 11:26:50 -03:00
Julia Medina 00e6470ed0 Fix and extend the documentation of the new Settings api 2014-06-10 11:26:38 -03:00
Daniel Graña 9830efb946 Merge pull request #697 from allait/master 
Add a note about reporting security issues
 2014-06-03 11:13:13 -03:00
Daniel Graña 7752074c51 Merge pull request #719 from Digenis/master 
document spider.closed() shortcut
 2014-06-03 10:01:15 -03:00
Daniel Graña 0467ad7719 Merge pull request #699 from allait/telnet-port 
Bind telnet console and webservice to 127.0.0.1 by default
 2014-06-03 09:51:16 -03:00
Daniel Graña 8a1c67fdaa Merge pull request #724 from rafallo/master 
item_dropped signal should pass response arg as item_scraped does
 2014-06-03 09:48:06 -03:00
Alexander Chekunkov fa5a67729b DOWNLOADER setting 2014-06-02 13:11:02 +03:00
Alexander Chekunkov 1fba64d34e DOWNLOADER setting 2014-06-02 13:05:22 +03:00
Rafal Jagoda ad6e449c63 Merge remote-tracking branch 'upstream/master' 2014-05-19 16:18:33 +01:00
Nikolaos-Digenis Karagiannis 24073a2624 document spider.closed() shortcut 2014-05-12 22:25:18 +03:00
Rendaw 8bdb6e2e3e Elaborated request priority value. 2014-05-07 19:14:45 +09:00
Carlos Rivera 946b854ddf grammatical issue 2014-05-06 15:41:59 -05:00
Rafal Jagoda 9d226e6266 add response arg to item_dropped signal handlers 
#710
 2014-05-02 14:42:01 +01:00
Alexander Chekunkov 4aa6179af7 added short RFPDupeFilter.request_fingerprint interface description 2014-04-26 15:46:53 +03:00
Alexander Chekunkov baaa077725 added note about RFPDupeFilter.request_fingerprint overriding to the settings documentation 2014-04-26 15:43:12 +03:00
Pablo Hoffman 3932877bcd signals doc: make argument order more consistent with code (although it doesn't matter in practice) 2014-04-25 16:03:32 -03:00
Pablo Hoffman eb07e09166 Merge pull request #663 from pawl/patch-1 
fixed typo
 2014-04-24 17:59:36 -04:00
Mikhail Korobov bdea071af3 DOC selectors.rst cleanup 2014-04-24 22:54:17 +06:00
ncp1113 5f18816428 for loops have to have a : at the end of the line 
changed 3 instances
 2014-04-24 22:50:21 +06:00
Daniel Graña 18412d75b5 Ammend example nesting selectors 2014-04-24 10:32:17 -03:00
Daniel Graña b4593c2ae7 document shortcuts in TextResponse class 2014-04-24 00:15:00 -03:00
Daniel Graña 1117687c47 update docs 2014-04-23 23:39:58 -03:00
Alexey Bezhan 89159779d0 Bind telnet console and webservice to 127.0.0.1 by default 2014-04-17 15:16:58 +01:00
Alexey Bezhan 6d6282da30 Add a note about reporting security issues 2014-04-17 14:41:21 +01:00
Mikhail Korobov 2d3803672b DOC use top-level shortcuts in docs 2014-04-15 01:09:35 +06:00
Julia Medina 80081054a2 Fix broken links in documentation 2014-04-09 18:57:52 -03:00
Julia Medina f0a0d0bc22 Ignore known broken links in docs linkcheck 2014-04-09 18:46:21 -03:00
Ana Sabina Uban 73109bf950 Fixed SgmlLinkExtractor constructor to properly handle both string and list parameters (attrs, tags, deny_extensions) 2014-03-27 16:12:00 +02:00
Paul Brown a1ee354609 fixed typo 2014-03-20 15:16:48 -05:00
tpeng fd7b40e892 add message when raise IngoreReques; fix item_scraped document 2014-03-18 15:23:25 +08:00
Julia Medina b9e2aad874 Doc for disabling download handler 2014-03-12 23:21:33 -03:00
Julia Medina ca1c1a82b5 FormRequest doc improvements 
Clickdata doc enhancements:
 * Fix xml attributes mention
 * nr is 0-indexed reference
 2014-03-12 12:34:50 -03:00
Julia Medina e29ab4d112 New doc: clickdata in Formrequest.from_response 
Documentation about:
 * clickdata parameter in Formrequest.from_response
 * nr attribute in clickdata dict
 * default behaviour when clickdata is None
 2014-03-12 06:43:50 -03:00
Nikolaos-Digenis Karagiannis 8ac9d40665 ItemLoader doc: missing args in replace_value() 2014-03-09 15:16:57 +02:00
Pablo Hoffman 0bc2cbaf09 Merge pull request #613 from breno/master 
Update scrapy command line doc with additional scrapy parse options
 2014-03-07 10:35:58 -08:00
Mikhail Korobov d8fb4a35db Merge pull request #627 from Digenis/master 
SpiderMW doc typo: SWP request, response
 2014-03-07 12:41:39 +05:00
Capi Etheriel 72b6c96d9a Running lucasdemarchi/codespell to fix typos in docs 2014-03-06 12:40:55 -03:00
Nikolaos-Digenis Karagiannis 4335420f40 SpiderMW doc typo: SWP request, response 2014-03-06 16:09:37 +02:00
Alexey Bezhan 210a0a6fe1 Fix some typos, whitespace and small errors in docs 2014-02-27 18:02:22 +00:00
Alexey Bezhan 2e59d77580 Clarify MapCompose documentation 2014-02-27 17:56:14 +00:00
Cash Costello c532e13dcd Added missing word in practices.rst 2014-02-27 13:08:36 -02:00
Pablo Hoffman 6d8b7b29ef remove unused setting: DOWNLOADER_DEBUG 2014-02-27 12:16:05 -02:00
Breno Colom 551c1c5eae Update scrapy command line doc with additional scrapy parse options 2014-02-24 18:10:51 -05:00
Pablo Hoffman 822adb1d5f Merge pull request #602 from Blender3D/email 
Added a mimetype parameter to `MailSender.send`
 2014-02-20 12:05:14 -08:00
Capi Etheriel 1ec1100b27 fixes dynamic itemclass example usage of type() 2014-02-18 20:59:05 -03:00
Nikita Nikishin a676017fe4 Fixed #441. 2014-02-18 03:50:43 -05:00
Paul Tremberth 41765ca18d DupeFilter: add setting for verbose logging + stats counter for filtered requests 2014-02-17 13:42:42 +01:00
Pablo Hoffman d24212c8e5 remove references to deprecated scrapy-developers list 2014-02-16 21:44:49 -02:00
Rolando Espinoza 28f946b05f DOC Use pipelines module name instead of pipieline following default project files. 2014-02-15 11:01:26 -04:00
Daniel Graña 180fc98cd8 Add 0.22.2 release notes 2014-02-14 15:45:44 -02:00
Daniel Graña 6ca49ce76a Merge pull request #594 from dangra/593-engineslots 
fix a reference to unexistent engine.slots.
 2014-02-14 09:39:30 -08:00
Daniel Graña b58285b635 fix a reference to unexistent engine.slots. closes #593 2014-02-14 15:31:05 -02:00
Nikolaos-Digenis Karagiannis 43a797e2f7 downloaderMW doc typo (spiderMW doc copy remnant) 2014-02-11 22:30:00 +02:00
tracicot b2f4b296df Correct typos 2014-02-10 11:46:23 -02:00
Daniel Graña 783d8e3ae1 Add 0.22.1 release notes 2014-02-08 17:09:04 -02:00
Paul Tremberth 57f30bcb04 Docs: 4-space indent for final spider example 2014-02-01 23:34:55 +01:00
Rolando Espinoza a6279fe95b DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm. 2014-01-30 11:53:42 -04:00
Rolando Espinoza 4255e12bc7 Updated the tutorial crawl output with latest output. 2014-01-23 18:18:56 -04:00
Rolando Espinoza 9aab9224cb Updated shell docs with the crawler reference and fixed the actual shell output. 
Also updated the shell example with a reproducible code example.
 2014-01-23 18:04:57 -04:00
Daniel Graña 677afe7e54 show ubuntu setup instructions as literal code 2014-01-20 16:22:53 -02:00
Daniel Graña ebfb5b7096 replace warning about updating package lists by a note on package upgrade 2014-01-20 15:18:34 -02:00
Daniel Graña 52c3ff9190 fix apt-get line 2014-01-20 14:39:26 -02:00
Daniel Graña eb73ddd301 Update Ubuntu installation instructions 2014-01-20 14:15:01 -02:00
stray-leone 7f30a671c3 modify the version of scrapy ubuntu package 
latest version is 0.22.
with scrapy-0.18, tutorial project provides error
relative issue : https://github.com/scrapy/scrapy/issues/511
 2014-01-20 10:24:34 -02:00
Daniel Graña 431f2d109f fix 0.22.0 release date 2014-01-17 17:54:01 -02:00
Mikhail Korobov dfd13f9941 fix typos in news.rst and remove (not released yet) header 2014-01-18 01:50:19 +06:00
Daniel Graña d1f1b074e1 Merge 0.22.0 release notes 2014-01-17 15:55:46 -02:00
Mikhail Korobov 7b7a1d8dfd Make Filesystem storage backend default again. See GH-500. 2014-01-17 04:32:08 +06:00
Mikhail Korobov b03fe04999 Rename AjaxCrawlableMiddleware to AjaxCrawlMiddleware 2014-01-16 23:09:37 +06:00
Pablo Hoffman ed6fd4933f Merge pull request #524 from hobsonlane/master 
documentation code example corrections per pablohoffman
 2014-01-16 06:44:51 -08:00
Pablo Hoffman 71ada5476e Merge pull request #472 from redapple/exslt 
Register EXSLT namespaces by default (resolves #470)
 2014-01-16 06:32:05 -08:00
Daniel Graña b9bb9bed6b Merge pull request #343 from kmike/ajax-crawlable 
[MRG] AjaxCrawlableMiddleware
 2014-01-16 05:07:39 -08:00
Hobson Lane 85a80d0752 remove "for brevity's sake" line and correct "Torrent item" 
Torrent item -> TorrentItem class
 2014-01-15 17:29:23 -08:00
Paul Tremberth 827c0cf51f Rename "regexp" prefix to "re" 2014-01-15 15:00:25 +01:00
Paul Tremberth 88c8a523a7 Add warning in docs on performance when using EXSLT regexp functions 2014-01-15 12:52:10 +01:00
Paul Tremberth a3eba68aca Drop EXSLT strings and math extensions 2014-01-15 12:28:25 +01:00
Pablo Hoffman ea2f897b81 Merge pull request #502 from scrapy/doc-fixes 
DOWNLOAD_DELAY docs clarification
 2014-01-14 21:07:42 -08:00
Hobson Lane a3db95985b another import name correction by pablo 2014-01-14 21:04:15 -08:00
Pablo Hoffman 3d308d7732 Merge pull request #517 from hoprocker/patch-1 
Clarify return value from extract_links
 2014-01-14 21:01:32 -08:00
Paul Tremberth 2cc26e6f56 Fix typo error 2014-01-14 13:09:18 +01:00
Paul Tremberth 29fc9f3466 Update selectors documentation and tests 2014-01-14 12:56:37 +01:00
Ferdy Rodriguez 807dd25324 fixed error on tor's name 2014-01-13 00:03:58 -06:00
Ferdy Rodriguez 8b9348cfaf Changed TOR Info as previous was removed from www.mininova.org 2014-01-12 23:46:04 -06:00
Hobson Lane 6ba0857a5c documentation code example correction corrections per pablohoffman 2014-01-10 10:37:27 -08:00
malcolm m 962e5ef702 Clarify return value from extract_links 2014-01-05 14:42:48 -08:00
Yuri Prezument 060891c01c Remove unused import from code sample 
Item pipeline docs - removed unused import from code sample
 2014-01-03 15:44:17 +02:00
Mikhail Korobov a27d91f0a6 Rename BaseSpider to Spider. See GH-495. 2013-12-30 19:46:41 +06:00
Mikhail Korobov e713733edf minor fixes to scrapy shell docs 
* better IPython links;
* MDC link instead of w3schools;
* small formatting fixes;
* show quoted URL in example
 2013-12-30 10:27:39 +06:00
Mikhail Korobov f18ac02987 remove duplicated link extractors link 
Check http://doc.scrapy.org/en/latest/topics/link-extractors.html - two menu items are highlighted at the left.
 2013-12-28 05:40:10 +05:00
Mikhail Korobov 9a999daa2a DOWNLOAD_DELAY docs clarification: 
* delay is enforced per website, not per spider;
* document download_delay attribute (it was previously documented only in FAQ about 999 error codes);
* document how CONCURRENT_REQUESTS_PER_IP affects download delays.
 2013-12-28 06:30:34 +06:00
Pablo Hoffman e42e3743fe quick documentation for #475 2013-12-24 12:19:15 -02:00
Mikhail Korobov e0cebbfc8f add a remark about 1% 2013-12-20 23:12:37 +06:00
Mikhail Korobov 943a0bd264 AjaxCrawlableMiddleware in Broad Crawl docs 2013-12-19 01:01:26 +06:00
Mikhail Korobov a87b3bd1c8 AjaxCrawlableMiddleware 2013-12-19 00:06:47 +06:00
RasPat1 ff21281b95 Note about selector class import 
This is the salient point of this code compared to the last example.  We have a selector now and this is how we use it.  Especially since the user has just come from the shell where the pre-instantiated selector is taken for granted.
 2013-12-15 13:46:42 -05:00
Daniel Graña 8a7c5b5d81 Add 0.20.2 release notes 
Conflicts:
	docs/news.rst
 2013-12-09 18:33:46 -02:00
Pablo Hoffman f2741c413e fix method name in tutorial. closes GH-480 2013-12-02 13:24:12 -02:00
Daniel Graña e34ffc0f42 Add 0.20.1 release notes 
Conflicts:
	docs/news.rst
 2013-11-28 16:25:57 -02:00