32423d4a33
some improvements to overview page
2015-03-25 19:27:52 -03:00
933dbc6be6
Oops
2015-03-25 18:33:17 -03:00
ca2575001e
Add missing callback arg in jobs topic example.
2015-03-25 18:32:20 -03:00
5ac91e4883
DOC remove Dynamic Creation of Item Classes section
...
It was a hack, and dicts-as-items cover most use cases.
Dicts don't allow to attach metadata to fields,
but e.g. adding "_meta" key and removing it in a custom serializer
is no worse than creating classes dynamically.
2015-03-23 18:11:35 +05:00
cda3922507
Add Response.urljoin() helper
2015-03-19 19:07:52 -03:00
c81eefaf81
fix doc links
2015-03-19 17:42:48 -03:00
8ac397670f
DOC move .. module: declaration to a proper place
2015-03-19 21:41:36 +05:00
643984e1b4
Updated architecture.rst
...
Added http://krondo.com/blog/?page_id=1327 as a resource
2015-03-18 23:55:22 -03:00
f16a33f34e
DOC change structure of spider docs:
...
* start with scrapy.Spider, then mention spider arguments,
then describe generic spiders;
* change wording regarding start_urls/start_requests;
* show an example of start_requests vs start_urls;
* show an example of dicts as items;
* as defining Item is an optional step now, docs for Items are
moved below Spider docs.
2015-03-19 05:25:15 +05:00
817dbc6cbd
DOC mention dicts in documentation; explain better what are Items for
2015-03-19 05:16:14 +05:00
959aaad205
Document `re_first`
2015-03-18 21:11:08 -03:00
127c6c694a
Fix extract_first() docs
2015-03-18 21:11:08 -03:00
012211accd
Add docs for extract_first()
2015-03-18 21:11:08 -03:00
12eedd90d1
fix truncated 0.24.5 release notes. closes #1084
2015-03-18 20:46:28 -03:00
4fb818a250
Run linkfix over current docs
2015-03-18 20:04:14 -03:00
ff987fb5a5
Add linkfix rule to docs Makefile
2015-03-18 20:00:03 -03:00
c05f5f175e
Added linkfix script to docs/utils
...
https://github.com/scrapy/scrapy/pull/1041#issuecomment-78143576
2015-03-18 19:41:26 -03:00
5a58d64131
Fix some redirection links in documentation
...
Fixes #606
2015-03-18 19:41:26 -03:00
ee82fe0e24
Merge pull request #1016 from SudShekhar/jsonProcessor
...
[MRG+1] Added JmesSelect
2015-03-18 08:11:25 -03:00
39635e5f55
Allow spiders to return dicts. See GH-1064.
2015-03-18 07:26:56 +05:00
934584a355
Merge pull request #1020 from jojje/gzip_http_cache
...
[MRG+1] add gzip compression to filesystem http cache backend
2015-03-17 14:32:06 -03:00
f924567591
Merge pull request #983 from ArturGaspar/linkextractor_css
...
[MRG+1] CSS support in link extractors
2015-03-17 01:07:47 -03:00
c13e23641b
httpcache dont_cache meta #19 #689
2015-03-16 11:50:04 -03:00
c1bf8365d5
Merge pull request #1022 from scrapy/docs-reorder
...
DOC reorder topics
2015-03-13 16:40:47 +05:00
baf5c59386
Merge pull request #1071 from eliasdorneles/updating-request-meta-special-keys
...
updating list of Request.meta special keys
2015-03-13 16:38:19 +05:00
57a5ee0097
added example value to set for proxy meta key
2015-03-12 23:20:44 -03:00
f7031c08ff
updating list of Request.meta special keys
2015-03-10 22:29:07 -03:00
c9d7386a32
Add 0.24.5 release notes
2015-02-25 19:36:40 -02:00
839ffba971
Added the first version of SelectJmes
...
Utilizes jmespath. Also, added tests and documentation for the same.
2015-02-24 22:59:01 +05:30
6419f76bbc
Merge pull request #1029 from dangra/contribute-to-master
...
DOC contribute to master branch
2015-02-03 02:10:47 +05:00
8a3b9b6131
Merge pull request #1011 from SudShekhar/master
...
Extension example fix to something that makes more sense
2015-01-30 15:45:52 -02:00
e42a1ac1a1
Reset items_scraped instead of item_count
...
items_scraped is the counter that needs to be reset each time we have scraped a specific number of items in the code instead of item_count (which represents the specific number of items needed before a message is logged). Updating the source code to reflect this.
Removed some irrelevant words from the log message.
Signed-off-by: Sudhanshu Shekhar <sudshekhar02@gmail.com>
2015-01-30 23:13:06 +05:30
f0bdc14522
Tentative attention message about what document to read for contributions
2015-01-29 15:40:12 -02:00
a07b4353d6
DOC reorder topics
...
* duplicate "topics/commands" link is removed;
* Request/Response docs are moved to "Basic Concepts";
* settings docs are moved to "Basic Concepts";
* exceptions docs are moved to "Basic Concepts";
* "signals" and "exporters" docs are moved to "Extending Scrapy";
* "Reference" section is dropped because it is empty now.
2015-01-22 22:58:10 +05:00
bd5d99a2d2
add gzip compression to filesystem http cache backend
2015-01-21 20:18:11 +01:00
4bc14da59e
Updates documentation on dynamic item classes.
...
Fixes #398
2015-01-19 17:21:56 -02:00
283d6a5344
DOC a couple more references are fixed
2015-01-19 22:07:03 +05:00
73e6b35622
DOC fix a reference
2015-01-19 22:02:46 +05:00
b0730a1d16
documentation for CSS support in link extractors
2014-12-11 18:22:08 -02:00
3602fc4fcb
fixed the variable types in mailsender documentation
2014-12-10 22:48:09 +01:00
e04b0aff74
An attempt to resolve #977 , add signal to be sent when request is dropped by the scheduler
2014-11-27 15:10:15 +03:00
a69f042d10
add 2 more test cases and minor doc fixes
2014-11-19 15:31:07 +01:00
fa84730e70
avoid download large response
...
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
2014-11-12 12:28:02 +01:00
b422312a38
Update docs copyright year range
2014-11-09 21:08:27 -05:00
13f83f0da0
typo
2014-11-10 06:28:41 +05:00
b21a28cc9a
Afterwords->Afterwards
2014-11-10 06:28:09 +05:00
2c67bd6c57
pywin32 is required by Twisted. closes #937
...
see:
* http://twistedmatrix.com/trac/ticket/6032
* https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2028
2014-11-05 23:05:54 -02:00
6cb8995731
Update install.rst
...
fixes #937
2014-11-05 22:50:52 -02:00
efe589c643
Merge pull request #882 from ahlen/feature/csvfeed-quotechar
...
[MRG+1] Allow to specify the quotechar in CSVFeedSpider
2014-11-04 11:32:59 -02:00
38dcf50cd6
comma instead of fullstop
2014-10-25 09:19:50 +06:00
675fd5ba04
Merge pull request #898 from scrapy/download-timeout
...
[MRG] DOC document download_timeout
2014-10-24 16:52:42 -02:00
0dce283459
Merge pull request #893 from kmike/less-ads
...
[MRG] DOC simplify extension docs
2014-10-21 17:13:59 -02:00
7d68b084a4
DOC document download_timeout Request.meta key and download_timeout spider attribute.
2014-10-07 04:23:11 +06:00
ea3b372b4f
DOC typo fix in leaks.rst
2014-10-02 15:20:13 +06:00
e7843d35de
Merge pull request #894 from kmike/leaks-docs
...
Leaks docs
2014-10-02 01:14:54 -03:00
5835224eee
Merge pull request #896 from scrapy/robotstxt-once
...
[MRG] process robots.txt once
2014-10-02 00:58:55 -03:00
6fcf9dce50
DOC document from_crawler method for item pipelines; add an example.
2014-09-25 03:13:51 +06:00
36eec8f413
dont_obey_robotstxt meta key; don't process requests to /robots.txt
2014-09-23 00:10:43 +06:00
bdbca1e2d7
DOC request queue memory usage
2014-09-21 07:30:44 +06:00
bc0f481a73
DOC bring back notes about multiple spiders per process because it is now documented how to do that
2014-09-21 07:12:01 +06:00
a122fdbfea
Update leaks.rst: there is now only a single spider in a process.
2014-09-21 06:54:00 +06:00
e435b3e3a3
DOC simplify extension docs
2014-09-21 00:19:24 +06:00
a312ebfb43
Update request-response.rst
...
Fixed minor typo
2014-09-14 22:06:31 +06:00
47b6dff9f1
Allow to specify the quotechar in CSVFeedSpider
2014-09-13 02:14:57 +02:00
16e62e9c9b
Per-spider settings documentation
2014-09-01 21:56:57 -03:00
ccde3317d7
Merge pull request #816 from Curita/api-cleanup
...
GSoC API cleanup
2014-09-01 21:55:36 -03:00
774ab74ad2
Merge pull request #864 from younghz/master
...
Duplicate comma in request-response.rst
2014-08-28 18:52:51 +06:00
d49766a6ac
Duplicate comma in request-response.rst
2014-08-28 19:58:58 +08:00
841dd5f1f5
Update webservice.rst
2014-08-18 17:48:01 -03:00
d684ecad7b
Merge pull request #846 from rocioar/master
...
fix dont_merge_cookies bad behaviour when set to false on meta
2014-08-18 13:54:11 -03:00
a9292cfab7
jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository
2014-08-15 23:28:13 -03:00
51b0bd281d
fix dont settings on meta behaviour, add docs and tests
2014-08-15 13:47:42 -07:00
3547ca6e61
Add example on running spiders outside projects
2014-08-14 11:50:33 -03:00
419026615f
Deprecate Crawler.spiders attribute
2014-08-14 09:19:41 -03:00
c90977ca98
Drop support for scrapy.project.crawler (And scrapy.stats consequently)
2014-08-12 14:02:56 -03:00
900a487682
Support multiple simultaneous LogObservers listening different crawlers
2014-08-12 14:02:56 -03:00
d40273561d
CrawlerProcess cleanup changes
2014-08-12 14:02:55 -03:00
980e30a187
Crawler interface cleanup
2014-08-12 14:02:55 -03:00
d7038b2a13
SpiderManager interface cleanup
2014-08-12 14:02:55 -03:00
39c6a80f9d
Both getdict and getlist return copies of the requested values
2014-08-12 14:02:55 -03:00
3ae971468f
Add Settings.copy, freeze and frozencopy method
2014-08-12 14:02:55 -03:00
84fa004793
Add from_crawler class method to base Spider
2014-08-11 11:23:57 -03:00
8fece4b0b8
Add 0.24.4 release notes
2014-08-09 17:21:20 -03:00
4badcc077d
Add 0.24.3 release notes
2014-08-09 00:31:26 -03:00
0772201ac8
Update installation docs
2014-08-08 18:07:38 -03:00
1a96cd5a3d
update tox usage on contributing.rst
2014-07-31 12:28:35 -03:00
242c08553f
move tests under root dir
2014-07-30 19:45:08 -03:00
b6b44a33f7
Update shell.rst typo
...
There was a typo.
"start" was mistyped as "star"
2014-07-28 12:26:13 -04:00
c298eaf164
removes weird indentation in the shell results
2014-07-23 20:42:36 -03:00
514e8582d0
improved explanations, clarified blog post as source, added link for XPath string functions in the spec
2014-07-23 19:43:57 -03:00
80de43ddab
adding some xpath tips to selectors docs
2014-07-22 23:20:53 -03:00
e441a01818
Updated input/ouput processor example according to #560 .
2014-07-16 09:04:48 -04:00
1a950e8f5a
Fix typo in commands.rst
2014-07-10 14:48:15 -03:00
379f6ee6b2
Add 0.24.2 release notes
2014-07-08 05:53:35 -03:00
0466e8cb7a
Fixed Python syntax in tutorial.
2014-07-04 10:38:01 +02:00
00cd4f0fa5
deprecate SgmlLinkExtractor
2014-07-02 23:15:17 +06:00
173a9dfe86
DOC fix formatting in release notes
2014-06-28 13:28:57 +06:00
284a2a06c9
Add 0.24.1 release notes
2014-06-27 12:03:37 -03:00
77c8284fa5
set 0.24.0 release date
2014-06-26 11:45:32 -03:00
ee33efac39
Add 0.24.0 release notes
2014-06-26 10:29:47 -03:00
436c1c8980
address latest comments
2014-06-25 15:12:58 -03:00
2ad8db6ae6
Merge pull request #761 from dangra/lxmlextractor
...
Promote LxmlLinkExtractor as LxmlExtractor
2014-06-25 15:07:02 -03:00
a9ecef5662
promote LxmlLinkExtractor as default in docs
2014-06-25 14:34:30 -03:00
5b2faf61c3
recognize jl extension as jsonlines exporter and update docs
2014-06-25 13:55:15 -03:00
1b32ece918
Merge branch 'settings-cleanup' of https://github.com/Curita/scrapy into Curita-settings-cleanup
2014-06-25 02:55:20 -03:00
499438ec44
indent parsed-literal as part of ordered list
2014-06-24 10:32:56 -03:00
ee5c3209af
doc: use |version| substitution in ubuntu packages
2014-06-24 09:52:51 -03:00
90e69141d3
Add doc on LxmlLinkExtractor class
2014-06-23 22:35:56 +02:00
05ffca2781
Merge pull request #626 from dangra/500-leveldb-cache
...
[WIP] Add a LevelDB cache backend
2014-06-23 15:30:13 -03:00
a90fd81c97
add leveldb cache backend docs
2014-06-23 14:37:29 -03:00
7fd8a1a387
minor change to request_scheduled signal doc
2014-06-14 20:51:16 -03:00
5f79940295
Document signal "request_scheduled"
2014-06-14 11:08:18 +03:00
ad11fc01cb
crate release notes for 0.24 and #699 to it
2014-06-10 18:46:53 -03:00
ed033854e2
setmodule helper method on Settings class
2014-06-10 11:26:50 -03:00
bdca06240c
Fix settings repr on the logs of the shell and tutorial docs topics
2014-06-10 11:26:50 -03:00
8a3a6236d9
Settings topic updated
2014-06-10 11:26:50 -03:00
00e6470ed0
Fix and extend the documentation of the new Settings api
2014-06-10 11:26:38 -03:00
9830efb946
Merge pull request #697 from allait/master
...
Add a note about reporting security issues
2014-06-03 11:13:13 -03:00
7752074c51
Merge pull request #719 from Digenis/master
...
document spider.closed() shortcut
2014-06-03 10:01:15 -03:00
0467ad7719
Merge pull request #699 from allait/telnet-port
...
Bind telnet console and webservice to 127.0.0.1 by default
2014-06-03 09:51:16 -03:00
8a1c67fdaa
Merge pull request #724 from rafallo/master
...
item_dropped signal should pass response arg as item_scraped does
2014-06-03 09:48:06 -03:00
fa5a67729b
DOWNLOADER setting
2014-06-02 13:11:02 +03:00
1fba64d34e
DOWNLOADER setting
2014-06-02 13:05:22 +03:00
ad6e449c63
Merge remote-tracking branch 'upstream/master'
2014-05-19 16:18:33 +01:00
24073a2624
document spider.closed() shortcut
2014-05-12 22:25:18 +03:00
8bdb6e2e3e
Elaborated request priority value.
2014-05-07 19:14:45 +09:00
946b854ddf
grammatical issue
2014-05-06 15:41:59 -05:00
9d226e6266
add response arg to item_dropped signal handlers
...
#710
2014-05-02 14:42:01 +01:00
4aa6179af7
added short RFPDupeFilter.request_fingerprint interface description
2014-04-26 15:46:53 +03:00
baaa077725
added note about RFPDupeFilter.request_fingerprint overriding to the settings documentation
2014-04-26 15:43:12 +03:00
3932877bcd
signals doc: make argument order more consistent with code (although it doesn't matter in practice)
2014-04-25 16:03:32 -03:00
eb07e09166
Merge pull request #663 from pawl/patch-1
...
fixed typo
2014-04-24 17:59:36 -04:00
bdea071af3
DOC selectors.rst cleanup
2014-04-24 22:54:17 +06:00
5f18816428
for loops have to have a : at the end of the line
...
changed 3 instances
2014-04-24 22:50:21 +06:00
18412d75b5
Ammend example nesting selectors
2014-04-24 10:32:17 -03:00
b4593c2ae7
document shortcuts in TextResponse class
2014-04-24 00:15:00 -03:00
1117687c47
update docs
2014-04-23 23:39:58 -03:00
89159779d0
Bind telnet console and webservice to 127.0.0.1 by default
2014-04-17 15:16:58 +01:00
6d6282da30
Add a note about reporting security issues
2014-04-17 14:41:21 +01:00
2d3803672b
DOC use top-level shortcuts in docs
2014-04-15 01:09:35 +06:00
80081054a2
Fix broken links in documentation
2014-04-09 18:57:52 -03:00
f0a0d0bc22
Ignore known broken links in docs linkcheck
2014-04-09 18:46:21 -03:00
73109bf950
Fixed SgmlLinkExtractor constructor to properly handle both string and list parameters (attrs, tags, deny_extensions)
2014-03-27 16:12:00 +02:00
a1ee354609
fixed typo
2014-03-20 15:16:48 -05:00
fd7b40e892
add message when raise IngoreReques; fix item_scraped document
2014-03-18 15:23:25 +08:00
b9e2aad874
Doc for disabling download handler
2014-03-12 23:21:33 -03:00
ca1c1a82b5
FormRequest doc improvements
...
Clickdata doc enhancements:
* Fix xml attributes mention
* nr is 0-indexed reference
2014-03-12 12:34:50 -03:00
e29ab4d112
New doc: clickdata in Formrequest.from_response
...
Documentation about:
* clickdata parameter in Formrequest.from_response
* nr attribute in clickdata dict
* default behaviour when clickdata is None
2014-03-12 06:43:50 -03:00
8ac9d40665
ItemLoader doc: missing args in replace_value()
2014-03-09 15:16:57 +02:00
0bc2cbaf09
Merge pull request #613 from breno/master
...
Update scrapy command line doc with additional scrapy parse options
2014-03-07 10:35:58 -08:00
d8fb4a35db
Merge pull request #627 from Digenis/master
...
SpiderMW doc typo: SWP request, response
2014-03-07 12:41:39 +05:00
72b6c96d9a
Running lucasdemarchi/codespell to fix typos in docs
2014-03-06 12:40:55 -03:00
4335420f40
SpiderMW doc typo: SWP request, response
2014-03-06 16:09:37 +02:00
210a0a6fe1
Fix some typos, whitespace and small errors in docs
2014-02-27 18:02:22 +00:00
2e59d77580
Clarify MapCompose documentation
2014-02-27 17:56:14 +00:00
c532e13dcd
Added missing word in practices.rst
2014-02-27 13:08:36 -02:00
6d8b7b29ef
remove unused setting: DOWNLOADER_DEBUG
2014-02-27 12:16:05 -02:00
551c1c5eae
Update scrapy command line doc with additional scrapy parse options
2014-02-24 18:10:51 -05:00
822adb1d5f
Merge pull request #602 from Blender3D/email
...
Added a mimetype parameter to `MailSender.send`
2014-02-20 12:05:14 -08:00
1ec1100b27
fixes dynamic itemclass example usage of type()
2014-02-18 20:59:05 -03:00
a676017fe4
Fixed #441 .
2014-02-18 03:50:43 -05:00
41765ca18d
DupeFilter: add setting for verbose logging + stats counter for filtered requests
2014-02-17 13:42:42 +01:00
d24212c8e5
remove references to deprecated scrapy-developers list
2014-02-16 21:44:49 -02:00
28f946b05f
DOC Use pipelines module name instead of pipieline following default project files.
2014-02-15 11:01:26 -04:00
180fc98cd8
Add 0.22.2 release notes
2014-02-14 15:45:44 -02:00
6ca49ce76a
Merge pull request #594 from dangra/593-engineslots
...
fix a reference to unexistent engine.slots.
2014-02-14 09:39:30 -08:00
b58285b635
fix a reference to unexistent engine.slots. closes #593
2014-02-14 15:31:05 -02:00
43a797e2f7
downloaderMW doc typo (spiderMW doc copy remnant)
2014-02-11 22:30:00 +02:00
b2f4b296df
Correct typos
2014-02-10 11:46:23 -02:00
783d8e3ae1
Add 0.22.1 release notes
2014-02-08 17:09:04 -02:00
57f30bcb04
Docs: 4-space indent for final spider example
2014-02-01 23:34:55 +01:00
a6279fe95b
DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm.
2014-01-30 11:53:42 -04:00
4255e12bc7
Updated the tutorial crawl output with latest output.
2014-01-23 18:18:56 -04:00
9aab9224cb
Updated shell docs with the crawler reference and fixed the actual shell output.
...
Also updated the shell example with a reproducible code example.
2014-01-23 18:04:57 -04:00
677afe7e54
show ubuntu setup instructions as literal code
2014-01-20 16:22:53 -02:00
ebfb5b7096
replace warning about updating package lists by a note on package upgrade
2014-01-20 15:18:34 -02:00
52c3ff9190
fix apt-get line
2014-01-20 14:39:26 -02:00
eb73ddd301
Update Ubuntu installation instructions
2014-01-20 14:15:01 -02:00
7f30a671c3
modify the version of scrapy ubuntu package
...
latest version is 0.22.
with scrapy-0.18, tutorial project provides error
relative issue : https://github.com/scrapy/scrapy/issues/511
2014-01-20 10:24:34 -02:00
431f2d109f
fix 0.22.0 release date
2014-01-17 17:54:01 -02:00
dfd13f9941
fix typos in news.rst and remove (not released yet) header
2014-01-18 01:50:19 +06:00
d1f1b074e1
Merge 0.22.0 release notes
2014-01-17 15:55:46 -02:00
7b7a1d8dfd
Make Filesystem storage backend default again. See GH-500.
2014-01-17 04:32:08 +06:00
b03fe04999
Rename AjaxCrawlableMiddleware to AjaxCrawlMiddleware
2014-01-16 23:09:37 +06:00
ed6fd4933f
Merge pull request #524 from hobsonlane/master
...
documentation code example corrections per pablohoffman
2014-01-16 06:44:51 -08:00
71ada5476e
Merge pull request #472 from redapple/exslt
...
Register EXSLT namespaces by default (resolves #470 )
2014-01-16 06:32:05 -08:00
b9bb9bed6b
Merge pull request #343 from kmike/ajax-crawlable
...
[MRG] AjaxCrawlableMiddleware
2014-01-16 05:07:39 -08:00
85a80d0752
remove "for brevity's sake" line and correct "Torrent item"
...
Torrent item -> TorrentItem class
2014-01-15 17:29:23 -08:00
827c0cf51f
Rename "regexp" prefix to "re"
2014-01-15 15:00:25 +01:00
88c8a523a7
Add warning in docs on performance when using EXSLT regexp functions
2014-01-15 12:52:10 +01:00
a3eba68aca
Drop EXSLT strings and math extensions
2014-01-15 12:28:25 +01:00
ea2f897b81
Merge pull request #502 from scrapy/doc-fixes
...
DOWNLOAD_DELAY docs clarification
2014-01-14 21:07:42 -08:00
a3db95985b
another import name correction by pablo
2014-01-14 21:04:15 -08:00
3d308d7732
Merge pull request #517 from hoprocker/patch-1
...
Clarify return value from extract_links
2014-01-14 21:01:32 -08:00
2cc26e6f56
Fix typo error
2014-01-14 13:09:18 +01:00
29fc9f3466
Update selectors documentation and tests
2014-01-14 12:56:37 +01:00
807dd25324
fixed error on tor's name
2014-01-13 00:03:58 -06:00
8b9348cfaf
Changed TOR Info as previous was removed from www.mininova.org
2014-01-12 23:46:04 -06:00
6ba0857a5c
documentation code example correction corrections per pablohoffman
2014-01-10 10:37:27 -08:00
962e5ef702
Clarify return value from extract_links
2014-01-05 14:42:48 -08:00
060891c01c
Remove unused import from code sample
...
Item pipeline docs - removed unused import from code sample
2014-01-03 15:44:17 +02:00
a27d91f0a6
Rename BaseSpider to Spider. See GH-495.
2013-12-30 19:46:41 +06:00
e713733edf
minor fixes to scrapy shell docs
...
* better IPython links;
* MDC link instead of w3schools;
* small formatting fixes;
* show quoted URL in example
2013-12-30 10:27:39 +06:00
f18ac02987
remove duplicated link extractors link
...
Check http://doc.scrapy.org/en/latest/topics/link-extractors.html - two menu items are highlighted at the left.
2013-12-28 05:40:10 +05:00
9a999daa2a
DOWNLOAD_DELAY docs clarification:
...
* delay is enforced per website, not per spider;
* document download_delay attribute (it was previously documented only in FAQ about 999 error codes);
* document how CONCURRENT_REQUESTS_PER_IP affects download delays.
2013-12-28 06:30:34 +06:00
e42e3743fe
quick documentation for #475
2013-12-24 12:19:15 -02:00
e0cebbfc8f
add a remark about 1%
2013-12-20 23:12:37 +06:00
943a0bd264
AjaxCrawlableMiddleware in Broad Crawl docs
2013-12-19 01:01:26 +06:00
a87b3bd1c8
AjaxCrawlableMiddleware
2013-12-19 00:06:47 +06:00
ff21281b95
Note about selector class import
...
This is the salient point of this code compared to the last example. We have a selector now and this is how we use it. Especially since the user has just come from the shell where the pre-instantiated selector is taken for granted.
2013-12-15 13:46:42 -05:00
8a7c5b5d81
Add 0.20.2 release notes
...
Conflicts:
docs/news.rst
2013-12-09 18:33:46 -02:00
f2741c413e
fix method name in tutorial. closes GH-480
2013-12-02 13:24:12 -02:00
e34ffc0f42
Add 0.20.1 release notes
...
Conflicts:
docs/news.rst
2013-11-28 16:25:57 -02:00
339861367e
Merge pull request #425 from audiodude/master
...
DownloaderMiddleware docs: Update process_request and minor cleanups.
2013-11-25 10:33:35 -08:00
14f5817d6b
Modify ItemLoader to support XPath and CSS selectors
...
Deprecate XPathItemLoader (now an alias to the new ItemLoader)
2013-11-21 18:05:24 +01:00
f87be371a2
better names for HANDLE_* settings, and added doc
2013-11-21 14:33:17 -02:00
e4c1d8d37d
Elaborate on use of order numbers
2013-11-19 17:51:50 -06:00
b878f60b5a
Add note to item-pipeline documentation explaining order in the ITEM_PIPELINES setting.
2013-11-19 16:12:54 -06:00
afe6eaa2fe
Merge pull request #460 from tntC4stl3/master
...
duplicate 'use' in line 87
2013-11-15 04:10:49 -08:00
b51d5d81e4
duplicate 'use' in line 87
2013-11-15 13:56:44 +08:00
04ff7ecebf
improve 0.20 release notes
...
Conflicts:
docs/news.rst
2013-11-08 17:45:03 -02:00
3d18a3c49e
bumped version to 0.21.0
2013-11-08 17:09:00 -02:00
d0980e5c9b
Merge 0.20 release notes
2013-11-08 17:06:10 -02:00
2df8156431
Drop Python 2.6 support
2013-10-29 13:44:00 -02:00
911c8082b0
simplified description of crawl command
2013-10-21 14:42:51 -02:00
e8ee449a2a
Merge pull request #432 from darkrho/crawl-url
...
Removed URL reference in crawl command and .tld suffix in docs for spider names
2013-10-21 09:40:58 -07:00
34543c2b2e
DOCS removed .tld suffix for spider names for the sake of consistency.
2013-10-19 23:03:20 -04:00
875b07aef8
fix references to old selector naming in docs
2013-10-17 09:33:15 -02:00
3043a5ba37
DownloaderMiddleware docs: Update process_request, proper explanation of IgnoreRequest.
...
Also:
* Change terminology to eliminate uses of terms such as "request middleware" to refer to the process_request methods of installed middleware.
* Remove description of "immediate redirection", as it is misleading.
Further changes.
2013-10-17 00:23:21 +00:00
086b8a20d4
typo fix in TextResponse docs
2013-10-17 04:50:30 +06:00
951a9f3f4c
Merge pull request #226 from scraperdragon/patch-1
...
Parameters to Request() in wrong order
2013-10-16 13:15:52 -07:00
1461363809
Replace `contenttype` references by `type`
...
The type to choose from is the selector type, not the input type. A
content-type doesn't make sense in this context.
2013-10-16 17:37:25 -02:00
155ea08ea1
use `sel` name for Selector's instances in docs, internals and shell
2013-10-15 15:58:42 -02:00
1abb1af0c6
fix typos and wording on selector's introduction
2013-10-15 10:13:43 -02:00
a3b711bdea
Move callback blob; mention errback
2013-10-15 12:19:42 +01:00
0ba0d85685
Parameters to Request() in wrong order
...
Implied that callback wasn't the first optional unnamed parameter.
2013-10-15 11:50:43 +01:00
28999590fa
update release notes
2013-10-14 16:41:04 -02:00
ab9462a251
remove more references to libxml2
2013-10-14 16:37:14 -02:00
4645f9e03c
Updates docs to reflect unified selectors api
2013-10-14 16:31:20 -02:00
bf37f78572
Drop libxml2 selectors backend
2013-10-11 18:02:35 -02:00
6d598f0d94
Update selectors docs
2013-10-10 18:24:00 -02:00
bc17e9d412
Adds HtmlCSSSelector and XmlCSSSelector classes, cssselect as optional dependency.
...
Ported .get() from _Element and .text_content() from HTMLMixin
Add CSS selectors to scrapy shell
Documenting CSS Selectors: Constructing selectors
Documenting CSS Selectors: Using Selectors
Make CSS Selectors a default feature.
Adds XPath powers to CSS Selectors and some syntactic sugar.
Removes methods copied over from lxml.html.HtmlMixin.
Updating docs to use new CSS Selector super powers.
Documenting CSS Selectors: Regular Expressions
Moving section after Nesting section, since it mentions it.
Documenting CSS Selectors: Nesting Selectors
Fix XPath specificity in lxml.selector.CSSSelectorMixin.text
Cleaning up unused stuff from cssel.py
Changing the behavior of lxml.selector.CSSSelectorMixin.text.
Concatenating all of the descendant text nodes is more useful
than returning it in pieces (there's xpath() if you need that).
Documenting CSS Selectors: CSS Selector objects
Documenting CSS Selectors: CSSSelectorList objects
Documenting CSS Selectors: HtmlCSSSelector objects
Documenting CSS Selectors: XmlCSSSelector objects
Fixing some documentations typos and errors
Enforcing the 80-char width lines
Tidying up CSS selectors and CSSSelectorMixin objects
Adding some missing references in documentation.
Fixing lxml.selector.CSSSelectorList.text
2013-10-10 18:23:15 -02:00
5b5dd679b0
Add 0.18.4 release notes
...
Conflicts:
docs/news.rst
2013-10-10 01:04:35 -02:00
e1683ddf9b
fix doc typo
2013-10-09 17:24:12 -02:00
8b9526a8f6
Merge pull request #400 from irgmedeiros/patch-2
...
Update the second code example
2013-10-07 07:57:18 -07:00
86c6e9433f
remove minor reference to 'scrapy server' command
2013-10-04 14:37:55 -03:00
aad90ec5a2
Add 0.18.3 release notes
...
Conflicts:
docs/news.rst
2013-10-03 12:56:25 -03:00
37c24e01d7
document bindaddress request meta
2013-10-02 17:13:17 -03:00
a9c3519897
updated required twisted version to 10.0
2013-10-01 14:07:38 -03:00
d6e3eae527
docs: added section regarding setting up django's settings.
2013-09-30 09:58:10 -04:00
0cc1d870db
docs: minor tidy up sample code and missing shell prompts.
2013-09-30 09:58:10 -04:00
8af0e89e85
Corrected typo.
2013-09-29 17:06:46 -04:00
f49f5724d5
Added dynamic creation of item classes to practices.rst.
2013-09-28 09:00:48 -04:00
9b50409986
Update the second code example
...
Update the second code example to reflect the last change in the first example.
2013-09-27 18:22:33 -03:00
d9e0fdc9aa
Update practices.rst
...
With this modification scrapy runs the spider with project settings. The previous example ran only with default settings resulting in ignoring all user settings as pipelines for example.
2013-09-27 17:56:30 -03:00
265910aae6
Merge pull request #363 from taikano/sitemap_alternate
...
also fetch alternate URLs from sitemaps, see #360
2013-09-26 09:15:02 -07:00
12280c2a95
fix sphinx references in doc
2013-09-25 15:13:17 -03:00
fc388f4636
Make ITEM_PIPELINE setting a dict
...
This is for consistency with how spider and downloader middlewares are
defined. ITEM_PIPELINE_BASE was also added and both remain empty.
Backwards compatibility is kept (with a warning) with list-based
ITEM_PIPELINES.
2013-09-23 17:50:43 -03:00
b1d1a36a1e
add note about enclosing urls with quotes when running from command-line. closes GH-384
2013-09-18 18:01:28 -03:00
71b320914a
Update request-response.rst
...
Fix small doc typo (too many backticks)
2013-09-18 11:45:25 -03:00
bbb0603091
Fixed directory location for dmoz_spider.py file
...
It should be under 'tutorial/spiders' not 'dmoz/spiders'
2013-09-15 21:55:52 -07:00
0400b18efa
docs: list lxml as installation prerequisite
2013-09-09 12:44:26 -03:00
6994959181
renamed to sitemap_alternate_links and added default value, see #360
2013-09-08 10:38:28 +02:00
8ed2d0cda1
improved changes to allow retrieval of alternate links in sitemaps, see #360
2013-09-07 12:56:30 +02:00
e6b3ca0180
Add 0.18.2 release notes
2013-09-03 14:29:40 -03:00
0f00b1602a
merge 0.18 release notes
2013-08-27 18:47:46 -03:00
19ff9ac4f9
url/body attributes of Request/Response objects are now immutable
2013-08-23 12:43:22 -03:00
86230c0ab8
added quantal & raring to support ubuntu releases
2013-08-22 21:49:55 -03:00
034ffae60f
Recommend Pillow instead of PIL. Closes GH-317.
2013-08-18 00:44:01 +06:00
32b6364bcd
#327 - Support STARTTLS / SSL option in email sender
2013-08-14 12:59:01 +02:00
c0b26e3d49
minor updates to 0.18 release notes
2013-08-14 01:39:44 -03:00
ed5b9068d2
fix contributters list format
2013-08-12 11:26:16 -03:00
a6693c9a5c
updated release notes and bumped version to 0.18.0
2013-08-09 19:02:28 -03:00
c00c4d7148
correction to description of example XPath retrieval in overview doc
2013-08-03 17:08:58 -07:00
0ad01c34d4
fixed typo to parallel fix on 0.16 branch
2013-08-03 17:06:10 -07:00
d227d530f6
Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware
...
Added COMPRESSION_ENABLE setting to docs
Added COMPRESSION_ENABLED setting to default settings
2013-08-01 11:31:28 -03:00
1ca31244b0
Fixed ordering of super argument call.
2013-07-16 14:50:10 -04:00
e12b689c4f
Updated documentation of spider arguments to include required super call.
2013-07-16 14:26:53 -04:00
1a1c93fafe
tiny FormRequest doc fix
2013-07-15 15:47:34 +06:00
ac2fadf3ab
DownloaderMiddleware.process_response docs fix
...
"returns an exception" -> "raises an exception"
2013-07-08 19:41:58 +06:00
39e5da5f66
improve docs for DownloaderMiddleware.process_response
2013-07-08 19:17:29 +06:00
0f4b70f582
remove no deprecated request_scheduled signal
...
It will be replaced by more accurate scheduler signals (proposal will
come soon)
2013-06-27 11:23:24 -03:00
bef8ade956
removed request_received and added request_scheduled
2013-06-26 16:45:46 -03:00
819b2776dd
Merge pull request #326 from berendiwema/master
...
Include example of how to stop the reactor from script
2013-06-25 13:30:07 -07:00
83b2774354
remove wrong default httpcache
2013-06-25 17:01:29 -03:00
aec314db09
added a bit more documentation on how to close the reactor when running scrapy from a script
2013-06-25 16:08:22 +02:00
bbde1d0e0b
Merge pull request #275 from stav/doc
...
doc: Response.replace() cannot take meta argument
2013-06-24 11:09:28 -07:00
50fa46d183
Document CrawlSpider.parse_start_urls method
2013-06-09 04:03:20 -03:00
b4fca90bba
merge 0.16.5 release notes
2013-05-30 18:49:00 -03:00
8007762890
Add FAQ entry referencing Request.meta usage
2013-05-27 13:02:17 -03:00
845c64b89d
add benchmarking to 0.18 release notes
2013-05-17 10:38:42 -03:00
ca12886acb
update copyright notes
2013-05-16 15:05:52 -03:00
8e49fed918
minor improvements to benchmarking doc
2013-05-16 13:23:13 -03:00
76087e336a
add scrapy bench command for benchmarking, with documentation
2013-05-16 13:15:25 -03:00
66311db23e
mention crawlera in best practices, as a way to deal with bans
2013-05-04 18:20:23 -03:00
9361c89573
remove scrapyd doc, as it was moved to its own repo
2013-04-27 04:15:42 -03:00
d02da2f31f
ported code to use queuelib
2013-04-23 17:48:09 -03:00
7a1536f76e
Merge pull request #290 from nramirezuy/item-copy
...
added copy method to item
2013-04-19 09:27:44 -07:00
6df274bba5
added copy method to item
2013-04-19 13:23:53 -03:00
b245d592aa
Update faq.rst
...
spider.DOWNLOAD_DELAY is deprecated
2013-04-18 02:42:15 +06:00
4de3aa4932
Update overview.rst
2013-04-08 14:13:15 +02:00
96c2332e0e
fix inaccurate downloader middleware documentation. refs #280
2013-04-02 11:35:32 -03:00
70179c7c0c
doc: remove trailing spaces
2013-03-21 13:57:39 -06:00
0d7747d353
doc: Response.replace() cannot take meta argument
...
>>> response.replace(meta={'foo':1})
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 45, in replace
return Response.replace(self, *args, **kwargs)
File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/__init__.py", line 77, in replace
return cls(*args, **kwargs)
File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 22, in __init__
super(TextResponse, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'meta'
2013-03-21 13:49:55 -06:00
2a5c7ed4da
make Crawler.start() return a deferred that is fired when the crawl is finished
2013-03-20 14:48:59 -03:00
b347c14b5f
update engine status output on telnet console documentation
2013-03-18 19:12:12 -03:00
5c2a82f1f7
fix typo
2013-03-17 19:34:55 +00:00
bb20907254
minor updated to faq
2013-03-14 16:43:00 -03:00
098ccff862
added FAQ about error: "cannot import name crawler"
2013-03-14 12:57:59 -03:00
8391b36251
minor updates to contributing doc
2013-03-13 03:24:25 -03:00
51c301b3a2
added link to python binary libs, for windows installation
2013-03-13 03:18:33 -03:00
8e72730792
Merge pull request #261 from stav/allowed_domains
...
allow spider allowed_domains to be set/tuple, #259
2013-03-12 20:44:51 -07:00
650eda68da
doc: add comment about commit history cleanliness
2013-03-10 18:51:04 -06:00
eeb69d2f70
added #260 to release notes
2013-03-08 11:59:38 -02:00
5b118ff4ab
added documentation (experimental feature)
2013-03-06 06:36:23 +11:00
3c8eef99cb
docs/contributing: added note explaining what Scrapy contrib is
2013-03-04 01:35:17 -02:00
f62b6660d4
doc: fix typo in spider middleware
2013-03-02 19:46:31 -06:00
7400ceb1ed
added 502 to RETRY_HTTP_CODES
2013-02-22 19:12:59 -02:00
a038f46859
doc: fixed rst title
2013-02-14 11:11:17 -02:00
22edc44c6c
doc: remove links to diveintopython.org, which is no longer available. closes #246
2013-02-14 11:09:40 -02:00
1ff8b4f831
updated release notes with previous commit
2013-02-12 00:59:25 -02:00
5db45b3825
remove scrapyd, it was migrated to its own repository
2013-02-06 05:24:07 +00:00
8e3b5baac5
Fix typo labeling attrs type bool instead of list
2013-02-05 15:10:41 -05:00
3cf7f4975b
Add 0.16.4 to release notes
...
Conflicts:
docs/news.rst
2013-01-23 11:29:38 -02:00
aae6aed4fb
fixes spelling errors in documentation
2013-01-22 14:52:18 -08:00
6ab8afb992
improve documentation about removing namespaces
2013-01-18 12:35:30 -02:00
1ba04b1fc3
added remove_namespaces() method to XmlXPathSelector objects
2013-01-18 12:20:03 -02:00
c31441a273
revert default HTTP cache policy to dummy (instead of RFC2616)
2013-01-17 13:08:29 -02:00
897195186a
document new FormRequest parameter named `formxpath` that matches forms using xpath
2013-01-08 18:36:20 -02:00
75563b3f00
Add list of supported and missing RFC2616 caching features
2013-01-08 18:16:44 -02:00
d8a760bf57
Merge branch 'http-cache-middleware'
...
Conflicts:
scrapy/contrib/downloadermiddleware/httpcache.py
scrapy/contrib/httpcache.py
scrapy/tests/test_downloadermiddleware_httpcache.py
2013-01-08 17:34:48 -02:00
864a7aef87
More httpcache updates
...
* Change default cache policy to RFC2616
* Update HttpCacheMiddleware documentation
* Move policies to scrapy.contrib.httpcache
* remove a lint error for .has_key() usage in DBM storage backend
2013-01-08 17:26:32 -02:00
672d09ea2e
add meta-refresh changes to release notes
2013-01-08 12:30:36 -02:00
defc4f89b5
update metarefresh settings
2013-01-08 11:41:19 -02:00
6a2b23883a
Add MetaRefreshMiddleware docs
2013-01-08 11:25:38 -02:00
076ba40404
update DOWNLOADER_MIDDLEWARES_BASE setting documentation
2013-01-08 10:50:27 -02:00
227a1d666b
add doc about disabling an extension. refs #132
2013-01-07 13:16:19 -02:00
5d3a4d755f
Update downloader middleware documentation
2013-01-06 18:53:14 +00:00
f9b130da12
Proposed Changes
...
I was very confused as to how you actually import DjangoItem.
I searched extensively on the internet looking for actual code so I could see how it worked.
I finally found http://blog.just2us.com/2012/07/setting-up-django-with-scrapy/ . It is much easier to understand with full files instead of code fragments.
I also edited where it says "we can see that the model is already saved" as I don't see how it's already saved.
2013-01-04 15:59:04 -05:00
d572f8945e
Fixed typo
...
'persitent' --> 'persistent'
2012-12-31 11:14:01 -08:00
492831fc6f
Merge branch 'master' of git://github.com/scrapy/scrapy into http-cache-middleware
2012-12-28 15:27:45 +01:00
93a1102189
Implemented policies for HTTP Cache
2012-12-26 16:29:48 -08:00
51b8feb4ce
fixed doc typos
2012-12-26 16:16:53 -02:00
1e2ee76df2
add documentation topics: Broad Crawls & Common Practies
2012-12-26 14:02:13 -02:00
fdaa35f6e8
Updated the downloader middleware documentation to reflect changes introduced by the support for real HTTP caching.
2012-12-24 19:37:53 +01:00
12475fccbe
Merge pull request #206 from dangra/downloader-enhancements
...
AutoThrottle and Downloader enhancements
2012-12-17 10:11:46 -08:00
d7daf836d5
Altering delay is enough to auto throttle downloads
2012-12-17 16:08:49 -02:00
5582ea28ec
Update docs/topics/commands.rst
...
A short change.
2012-12-10 15:16:02 -02:00
0cc138c010
Add 0.16.3 release notes
2012-12-07 18:55:15 -02:00
39274a2457
doc: removed obsolete references to ClientForm
2012-11-23 19:06:47 -02:00
99f164fc87
correct docs for default storage backend
2012-11-22 14:05:47 -06:00
1f0d167037
doc: removed broken proxyhub link from FAQ
2012-11-22 15:10:26 -02:00
097aea04a4
Fixed docs typo in SpiderOpenCloseLogging example
2012-11-10 12:24:53 +04:00
da7e414fe9
Add 0.16.2 release notes
...
Conflicts:
docs/news.rst
2012-11-09 13:03:04 -02:00
db21bccf9a
added 0.18 to release notes and mention spider contracts
2012-11-07 16:02:18 -02:00
aa0e02dc54
added open_in_browser to debugging doc
2012-11-04 19:58:06 -02:00
7a7c5d1334
removed reference to global scrapy stats from settings doc
2012-11-03 17:05:01 -02:00
c0542838d3
update news file with 0.16.1 release notes
2012-10-26 18:53:59 -02:00
8f4c879b58
extended documentation on how to access crawler stats from extensions
2012-10-25 11:28:23 -02:00
a9616f387e
fix dashes under rst headers
2012-10-18 20:14:30 -02:00
4852c6102d
set release date for 0.16.0 in news
2012-10-18 20:07:36 -02:00
1a905d62f5
removed scrapy.log.started attribute, and avoid checking if log has already been started (since it should be called once anyway)
2012-10-09 16:05:19 -02:00
1f89eb59fe
fixed doc reference to topics-contracts
2012-10-09 16:02:12 -02:00
9e4c6aba2f
updated release notes with new stats recently added
2012-10-09 12:50:42 -02:00
7458092eef
added spider contracts to release notes and warn that its API is still subject to change
2012-09-29 03:06:30 -03:00
c380910b40
Merge pull request #167 from alexcepoi/sep-017
...
Spider contracts (SEP-017)
2012-09-28 13:57:07 -07:00
a5f8943d3e
added change introduced in previous commit to release notes
2012-09-26 13:17:01 -03:00
11d29c7005
SEP-017 contracts: add tests and minor improvements
2012-09-21 00:12:46 +02:00
b46b5a6ef0
Documented AutoThrottle extension and added to extensions available by
...
default. Also deprecated concurrency and delay settings, in favour of
using the standard Scrapy ones.
2012-09-20 18:52:57 -03:00
c7f8219901
- removed scrapy.conf singleton from scrapy.log, scrapy.responsetypes,
...
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
2012-09-19 03:03:34 -03:00
303e13f616
selector documentation typos
2012-09-18 12:56:52 -05:00
3d736e657f
fixed typo in doc
2012-09-18 10:51:01 -03:00
eed6eb49da
make DBM the new default storage backend for HTTP cache middleware, simplified DBM storage backend code to avoid dealing with many spiders at once (not needed), and update httpcache stats names (hit -> hits, miss -> misses)
2012-09-17 10:11:07 -03:00
8f2dda12cc
removed another instance of scrapy.conf.settings singleton, this time from scrapy.utils.trackref. From now on, trackrefs functionality will be always enabled as it imposes a very minimal performance overhead
2012-09-16 21:21:44 -03:00
9685c24059
restored stats_spider_{opened,closing,closed} signals in backwards compatibility mode
2012-09-14 12:38:00 -03:00
81ed2d2d0b
major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (stats_spider_opened, etc). Stats are much simpler now, backwards compatibility is kept on the Stats Collector API.
2012-09-14 12:31:33 -03:00
d6867f7984
removed sphinx warnings about duplicate reference names 'this page'
2012-09-13 15:26:45 -03:00
a874964ad4
renamed 'XPath Selectors' title to just 'Selectors'
2012-09-13 15:24:44 -03:00
acb8895e1a
changed note in scrapyd doc to use sphinx notes
2012-09-13 15:22:59 -03:00
26f1d5cb48
Merge pull request #171 from artem-dev/scrapyd_job_times
...
added start and stop times for scrapyd list jobs web service
2012-09-13 11:16:01 -07:00
4d8f253912
commited doc file missed from prev commit
2012-09-12 11:12:59 +03:00
7ef593c5c2
refactored MailSender to get rid of scrapy.conf singleton, also removed ill-designed scrapy.mail.mail_sent signal
2012-09-11 16:27:19 -03:00
bf8dc61fb7
SEP-017 contracts: pretty-printing and docs
2012-09-10 23:17:27 +02:00
fff2871828
added doc section (and FAQ) about spider arguments
2012-09-04 14:49:30 -03:00
f4a17ec272
removed references to Scrapy Snippets site
2012-09-03 22:19:15 -03:00
b901e64044
replaced memory usage acounting with (more portable) resource module, removed scrapy.utils.memory module. closes #161
2012-09-03 19:28:16 -03:00
e2f9daac67
fixed formatting in scrapyd release notes
2012-09-03 16:58:58 -03:00
dd1398b280
removed another instance of scrapy.conf.settings singleton, this time from download handlers
2012-09-03 16:44:18 -03:00
7c8af83138
added issue role to documentation
2012-08-31 18:39:30 -03:00
a2d22307ad
update news.rst
2012-08-31 17:38:40 -03:00
be206ca5ab
added process_start_requests method to spider middlewares
2012-08-31 16:41:50 -03:00
b920674666
corrected minor issue with doc references
2012-08-31 16:41:49 -03:00
4ec99117d3
fixed minor doc typo
2012-08-30 11:56:30 -03:00
4a5f70278f
minor tidy up to installation guide windows notes
2012-08-29 15:44:24 -03:00
098d892c03
simplified installation guide to only mention pip/easy_install mechanism, and provide hints for Windows users
2012-08-29 15:37:05 -03:00
70f8e517a1
promoted DjangoItem to main contrib
2012-08-29 11:23:11 -03:00
da234af456
Merge branch 'singleton_removal'
2012-08-28 18:42:33 -03:00
babfc6e79b
Updated documentation after singleton removal changes.
...
Also removed some unused code and made some minor additional
refactoring.
2012-08-28 18:35:57 -03:00
5fa5c4545b
use ScrapyJSONEncoder JsonItemExporter & JsonLinesItemExporter, to support nested items properly
2012-08-22 13:46:50 -03:00
36f47a4aec
Removed per-spider settings concept, and scrapy.conf.settings singleton from many extensions and middlewares. There are some still remaining, that will be removed in future commits
2012-08-21 17:27:45 -03:00
abcc8c9f63
Recommend pypi as single way to install on Windows
2012-08-06 10:21:13 -03:00
b7b0c49520
append parse command to example code sections in docs. closes #162
2012-08-06 09:10:16 -03:00
832e45073b
fixed typo in stats documentation. closes #159
2012-07-20 17:13:06 -03:00
00bfb37e79
Update master
2012-07-04 06:55:01 +03:00
277ed0ae23
Merge pull request #145 from alexcepoi/cookies-changes
...
domain and path support for request cookies
2012-06-25 11:29:04 -07:00
177c81745d
domain and path support for request cookies
2012-06-25 20:17:59 +02:00
179e3810dc
fixed links to doc. closes #150
2012-06-24 01:00:33 -03:00
f4faa19e31
added docs topic debugging spiders
2012-06-21 20:03:33 +02:00
3e05a2ecf6
update docs for parse command
2012-06-12 18:28:10 +02:00
e1be9c01bc
updated FAQ about bot bans
2012-06-08 18:33:53 -03:00
ae2ff4d33a
update news file with 0.14.4 release notes
2012-05-15 16:16:02 -03:00
9686f97242
added precise to supported ubuntu distros
2012-05-12 19:54:36 -03:00
58e88ed246
scrapyd: do not set SCRAPY_FEED_URI/SCRAPY_LOG_FILE if items_dir/logs_dir settings are not set
2012-05-08 17:43:00 -03:00
9c3b9f2968
fixed bug in json-rpc webservice reported in https://groups.google.com/d/topic/scrapy-users/qgVBmFybNAQ/discussion . also removed no longer supported 'run' command from extras/scrapy-ws.py
2012-05-03 12:05:40 -03:00
abcac4fcbd
updated maintainer to scrapinghub
2012-05-02 03:25:35 -03:00
86dba76d1f
documentation indentation
2012-04-30 13:09:34 -05:00
c7f33c534b
removed redundant lines from release notes
2012-04-20 09:09:55 -03:00
15c8c01828
add 0.14.3 release notes
2012-04-19 23:25:15 -03:00
97f362d64e
removed deprecated/undocumented class: HTMLImageLinkExtractor
2012-04-19 12:07:46 -03:00
2fb5e62c39
doc: update overview page to point to the genspider command. refs #107
2012-04-19 02:37:22 -03:00
d567d8efbe
added note to docs/topics/firebug.rst about google directory being shut down
2012-04-19 01:34:20 -03:00
30ddbf624e
mention about some scrapy.xlib modules removed in the release notes
2012-04-17 12:31:18 -03:00
d99ee6deb9
added some missing entries to release notes
2012-04-17 12:29:48 -03:00
d8ebf16fe5
Merge pull request #114 from stav/master
...
Scrapy DOC changes
2012-04-11 12:08:45 -07:00
7cca916ed5
added release notes to official documentation, including all release notes since Scrapy 0.7
2012-04-11 15:53:23 -03:00
f1802289cd
small doc typo change to get the fork rolling
2012-04-11 12:05:39 -05:00
4f28ffcb2c
removed no longer needed dependency on simplejson
2012-04-10 16:01:36 -03:00
6e8edbd72e
switched default selectors backend to lxml
2012-04-10 15:52:14 -03:00
27018fced7
changed default user agent to Scrapy/0.15 (+ http://scrapy.org ) and removed no longer needed BOT_VERSION setting
2012-03-23 13:45:21 -03:00
8933e2f2be
added REFERER_ENABLED setting, to control referer middleware
2012-03-22 16:35:14 -03:00
da826aa13d
fixed minor mistake in Request objects documentation
2012-03-21 10:25:41 +08:00
175c70ad44
fixed minor defect in link extractors documentation
2012-03-20 22:56:45 -03:00
35fb01156e
removed some obsolete remaining code related to sqlite support in scrapy
2012-03-16 11:55:55 -03:00
b6ae266546
Removed (very old and possibly broken) backwards compatibility support for Twisted 2.5
2012-03-15 00:28:24 -03:00
e521da2e2f
Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25
2012-03-01 08:18:12 -02:00
2b16ebdc11
added minor clarification on cookiejar request meta key usage
2012-02-29 07:19:01 -02:00
5afe4f50c1
scrapyd: support bind to a specific ip address
2012-02-29 13:47:40 +08:00
81abb45000
fixed bug in new cookiejar documentation
2012-02-28 11:08:25 -02:00
26c8004125
added documentation for the new cookiejar Request.meta key
2012-02-27 19:58:58 -02:00
7fe7c3f3b1
MemoryUsage extension: close the spiders (instead of stopping the engine) when the limit is exceeded, providing a descriptive reason for the close. Also fixed default value of MEMUSAGE_ENABLED setting to match the documentation.
2012-02-23 17:05:06 -02:00
7b8942a648
updated StackTraceDump extension doc
2012-02-16 15:14:17 -02:00
ea77342b55
updated versioning doc according to recent changes
2012-01-05 11:50:28 -02:00
0b0bce7f3c
scrapyd: added cancel.json and listjobs.json api methods to documentation
2012-01-05 11:23:25 -02:00
8f42633a94
scrapyd: added clarification about how to disable items feeds generation
2012-01-05 11:20:50 -02:00
dbda33efa6
scrapyd: added support for storing items by default
...
Items are stored the same way as logs, in jsonlines format.
Also renamed logs_to_keep setting to jobs_to_keep.
2012-01-03 23:08:54 -02:00
0be421fbf0
fixed reference to tutorial directory
2011-12-23 18:57:11 -02:00
41fd3c4f6c
doc: removed duplicated callback argument from Request.replace()
2011-12-23 15:55:46 -02:00
0eeff76227
fixed formatting of scrapyd doc
2011-12-20 03:18:37 -02:00
bcb31988f2
change tutorial to follow changes on dmoz site
2011-12-14 13:03:31 -02:00
992af8d38f
ubuntu repos: added support for oneiric release
2011-10-25 14:26:38 -02:00
c38c49d56a
fixed PickeItemExporter bug, added unittest, and added pickle to suported feed exports formats
2011-10-25 02:36:51 -02:00
8bdf288428
made scrapyd doc more version agnostic
2011-10-23 05:29:54 -02:00
ade5efdc61
added -o option to scrapy crawl, a convenient shortcut for using feed exports
2011-10-22 20:53:49 -02:00
431441cb52
updated documentation to remove references to old issue tracker and mercurial repos
2011-09-25 13:06:24 -03:00
ce03ccd4ec
updated documentation about DEPTH_PRIORITY and DFO/BFO crawls
2011-09-23 13:22:25 -03:00
b7c436343a
scrapy deploy support git version
2011-09-21 22:17:08 +02:00
ab1c9cfc56
removed documentation header notifying about other documentation versions, as that's provided by readthedocs already
2011-09-14 02:39:32 -03:00
5f1b1c05f8
Do not filter requests with dont_filter attribute set in OffsiteMiddleware
2011-09-08 15:18:10 -03:00
bff3d31469
scrapyd: updated schedule.json response format
2011-09-04 09:29:24 -03:00
a1dbc62b45
removed CONCURRENT_SPIDERS setting (use scrapyd maxproc instead)
2011-09-02 18:27:39 -03:00
40f7075f11
added initial documentation about suspend and resume crawls
2011-09-02 13:12:27 -03:00
27dd68a690
added SpiderState extension
2011-09-02 13:06:59 -03:00
6a31ab667d
minor fix to doc
2011-09-01 15:08:23 -03:00
d98b058c21
no longer recommend using labmda's in the doc, as they're not friendly with scheduler persistence
2011-09-01 15:06:49 -03:00
76af0cdd44
updated documentation and code to use -s instead of --set option
2011-09-01 14:35:37 -03:00
98b68ca89d
scrapyd: documented support for passing setting to spiders in schedule.json
2011-08-27 01:31:12 -03:00
5c6b0631e2
minor doc fix
2011-08-19 11:42:03 -03:00
9d97e73a24
fixed priority handling on the new scheduler so that it's backwards compatible (ie. bigger priorities are higher). also fixed a few documentation bugs related to requests priority
2011-08-19 08:26:41 -03:00
a3697421c0
some minor updates to documentation
2011-08-11 09:19:59 -03:00
5da6ffb57b
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
2011-08-11 09:11:19 -03:00
bc2d2183e9
fixed import in doc
2011-08-11 09:11:08 -03:00
19e6da59d8
added new downloader middleware: ChunkedTransferMiddleware
2011-08-09 03:03:25 -03:00
984be35461
Some telnet console changes:
...
* renamed manager alias to crawler
* added aliases: spider, slot
* fixed est() function
2011-08-08 15:01:08 -03:00
f7c0aeccc6
added note about engine_started signal
2011-08-07 03:57:09 -03:00
9f60c27612
added setting to support disabling DNS cache: DNSCACHE_ENABLED
2011-08-05 20:41:59 -03:00
cb95d7a5af
added marshal to formats supported by feed exports
2011-08-03 16:16:48 -03:00
549725215e
Initial support for a persistent scheduler, to support pausing and resuming
...
crawls.
* requests are serialized (using marshal by default) and stored on disk, using
one queue per priority
* request priorities must be integers now
* breadh-first and depth-first crawling orders can now be configured
through a new DEPTH_PRIORITY setting (see doc). backwards compatilibty with
SCHEDULER_ORDER was kept.
* requests that can't be serialized (for example, non serializable callbacks)
are always kept in memory queues
* adapted crawl spider to work with persitent scheduler
2011-08-02 11:57:55 -03:00
f354a49d0f
added FAQ about preventing bots getting banned
2011-07-28 00:40:30 -03:00
ce7a787970
Big downloader refactoring to support real concurrency limits per domain/ip,
...
instead of global limits per spider which were a bit useless.
This removes the setting CONCURRENT_REQUESTS_PER_SPIDER and adds thre new
settings:
* CONCURRENT_REQUESTS
* CONCURRENT_REQUESTS_PER_DOMAIN
* CONCURRENT_REQUESTS_PER_IP (overrides per domain)
The AutoThrottle extension had to be disabled, but will be ported and
re-enabled soon.
2011-07-27 13:38:09 -03:00
c59340150f
Added cached DNS resolver based on old caching resolver extension from scrapy.contrib.resolver. This new one is *not* an extension, it comes builtin and always enabled.
2011-07-27 03:45:15 -03:00
2ac08a713d
downloader: renamed SpiderInfo to Slot, for consistency with engine and scraper names
2011-07-22 02:06:10 -03:00
0e008268e1
removed SimpledbStatsCollector from scrapy code, it was moved to https://github.com/scrapinghub/scaws
2011-07-20 10:38:16 -03:00
b6b0a54d9f
removed FAQ entry
2011-07-20 01:31:36 -03:00
e3f640c7bf
added FAQ entry about scrapy deploy issue on Mac + Python 2.5
2011-07-19 19:53:32 -03:00
84f518fc5e
More core changes:
...
* removed execution queue (replaced by newer spider queues)
* added real support for returning iterators in Spider.start_requests()
* removed support for passing urls to 'scrapy crawl' command
2011-07-15 15:18:39 -03:00
dbad1373f1
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
2011-07-13 18:44:54 -03:00
18cb4ff1d8
added natty to list of supporte ubuntu distros
2011-07-13 18:43:52 -03:00
39a2ea97c8
redirect mw: added REDIRECT_ENABLED setting and documented the other settings
2011-07-13 14:18:15 -03:00
541ed3913b
retry middleware: added RETRY_ENABLED setting and documented the other settings more properly, also improved messages when no longer retrying requests
2011-07-13 11:55:05 -03:00
763f3dc628
minor update to doc
2011-07-12 19:56:39 -03:00
bfda9ec319
added clarification about scrapy versioning including the recently adopted odd/even versioning scheme
...
--HG--
rename : docs/api-stability.rst => docs/versioning.rst
2011-07-12 19:53:23 -03:00
4fde1ef94d
added CloseSpider exception, to manually close spiders
2011-07-12 14:24:10 -03:00
db5cae7c03
SitemapSpider: added support for filtering which sitemaps to follow (patch contributed by Rolando Espinoza). closes #330
2011-06-23 18:18:29 -03:00
57c43fdce6
added SitemapSpider, with tests and doc
2011-06-15 11:54:34 -03:00
91dc46539f
added LogStats extension for periodically logging basic stats (like crawled pages and scraped items)
2011-06-14 00:50:05 -03:00
841e9913db
renamed CLOSESPIDER_ITEMPASSED setting to CLOSESPIDER_ITEMCOUNT, to follow the refactoring done in r2630
2011-06-13 16:58:51 -03:00
474cba512c
simplified MemoryDebugger extension to use stats for dumping memory debugging info
2011-06-06 03:13:28 -03:00
5fbc32c015
call stats collector engine_stopped() after the engine is closed (to make sure all data from extensions has been collected), and added that method to documented api
2011-06-06 03:12:40 -03:00
9d9c8877da
added 'scrapy edit' command
2011-06-05 22:02:56 -03:00
03ae481cad
removed experimental crawlspider v2
2011-06-03 18:23:23 -03:00
5bf733b6f6
Changed default representation of items to pretty-printed dicts. This improves
...
default logging by making log more readable in the default case, for both Scraped and Dropped lines.
Projects can still customize how items are represented by overriding the item's __str__ method, as usual.
2011-06-03 01:13:01 -03:00
1bc2339bb8
Merged item passed and item scraped concepts, as they have often proved
...
confusing in the past.
This means:
* original item_scraped signal was removed
* original item_passed signal was renamed to item_scraped
* old log lines "Scraped Item..." removed
* old log lines "Passed Item..." renamed to "Scraped Item..."
2011-06-03 01:13:00 -03:00
e6091df551
fixed doc typo
2011-05-30 09:04:31 -03:00
1d98fc8fb5
added spider_error signal
2011-05-29 22:38:17 -03:00
2fa0f75f2d
added COOKIES_ENABLED setting to support disabling the cookies middleware
2011-05-27 00:35:34 -03:00
d72d3f4607
stack trace dump extension: also dump engine status, and support triggering it with SIGQUIT, besides SIGUSR2
2011-05-20 03:25:00 -03:00
951ba507f9
Removed support for default values in Scrapy items, which have proven confusing in the past
2011-05-19 21:42:46 -03:00
503f302010
removed remaining references to scheduler middleware from doc, as it will be removed on next release
2011-05-18 19:48:48 -03:00
3fd17432cf
fixed outdated documentation
2011-05-18 14:46:20 -03:00
9016e7e993
added role to link to scrapy source code (not yet used)
2011-05-18 14:43:34 -03:00
cd85c12c33
Some Link extractor improvements:
...
* added support for ignoring common file extensions that are not followed if
they occur in links
* fixed link extractor documentation issues
* slighly improved performance of applying filters
* added link to link extractors doc from documentation index
2011-05-18 12:32:34 -03:00
495152bd50
disabled verbose depth stats collection by default, added DEPTH_STATS_VERBOSE setting to enable it
2011-05-18 11:04:48 -03:00
accb6ed830
dump stats to log by default (ie. change default value of STATS_DUMP to True)
2011-05-17 22:42:05 -03:00
7f97259ba7
added w3lib to requirements, in installation guide
2011-05-01 11:14:57 -03:00
4a83167698
fixed small doc typo
2011-04-30 01:35:30 -03:00
bb2b67c862
updated tutorial to use 'dmoz' as the name of the spider instead of 'dmoz.org', so that it's more similar to the dirbot example project
2011-04-28 09:31:57 -03:00
bf73002428
removed googledir example, replaced by dirbot project on github. updated docs accordingly
2011-04-28 02:28:39 -03:00
b12dd76bb8
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
2011-04-25 09:31:18 -03:00
678f08bc1b
added warning about using 'parse' as callback in crawl spider rules
2011-04-25 09:30:42 -03:00
ad496eb3b6
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
2011-04-14 12:36:27 -03:00
ecb4f44cbc
Added clarification on how to work with local settings and scrapy deploy
2011-04-14 12:36:09 -03:00
3ee2c94e93
Improved cookies middleware by making COOKIES_DEBUG nicer and documenting it
2011-04-06 14:54:48 -03:00
8a5c08a6bc
added join_multivalued parameter to CsvItemExporter
2011-03-24 13:15:52 -03:00
3954e600ca
added DBM storage backend for HTTP cache
2011-03-23 21:32:02 -03:00
cfd11df539
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
2011-02-24 15:28:57 -02:00
8f7e163b04
Fixed wrong method name in downloader middleware documentation
2011-02-24 15:26:32 -02:00
c55355642c
fix FAQ typos reported by marlun_ at #scrapy IRC channel
2011-02-16 08:57:42 -02:00
1fb55bdaf0
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
2011-02-15 07:25:12 -02:00
16d9a33951
added FAQ entry about working with big data feeds
2011-02-15 07:24:52 -02:00
936353d5f1
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
2011-02-09 11:20:46 -02:00
181d1c09ae
Fixed typo and code indentation in the doc. Closes #307 and #308
2011-02-09 11:19:46 -02:00
c91f0d9ea1
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
2011-02-04 13:39:54 -02:00
c5499ead73
Clarified behaviour when multiple rules match the same link in CrawlSpider
2011-02-04 13:39:12 -02:00
d7f193cbea
bumped version to 0.13 in documentation
2011-01-02 17:29:43 -02:00
b56e933be9
bumped version to 0.12 in documentation
2011-01-02 17:28:33 -02:00
5879389ad0
Bumped version to 0.12
2011-01-02 16:16:40 -02:00
fa644f7a5e
Some simplifications to Scrapyd architecture and internals:
...
- launcher no longer knows about egg storage
- removed get_spider_list_from_eggifile() file and replaced by simpler
get_spider_list() which doesn't receive en egg file as argument
- changed "egg runner" name to just "runner" to reflect the fact that it
doesn't necesarilly run eggs (though it does in the default case)
--HG--
rename : scrapyd/eggrunner.py => scrapyd/runner.py
2010-12-27 16:22:32 -02:00
633ebc4c43
minor indentation improvement
2010-12-23 13:04:49 -02:00
db07a9a938
Added notice to documentation, pointing dev to stable versions and viceversa
2010-12-23 13:03:40 -02:00
544308d6d0
updated ubuntu repos doc, in preparation for the 0.11 release
2010-12-21 11:02:56 -02:00
002abf204f
Updated item_passed signal to send passed item in 'item' argument, instead of 'output' argument, keeping backwards compatibility for the 'output' argument. Closes #273
2010-12-13 14:05:47 -02:00
f984d438a0
updated docs to use scrapy version on aptitude install lines
2010-12-13 14:02:42 -02:00
119fd20e91
Added verbose option to 'version' command. Closes #298
2010-12-13 00:32:44 -02:00
6a1b69c93f
renamed command 'scrapyd' to 'server', and deprecated 'runserver' and 'queue' commands
...
--HG--
rename : scrapy/commands/scrapyd.py => scrapy/commands/server.py
2010-11-30 20:23:27 -02:00
df54ed0041
Some Scrapyd enhancements:
...
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
2010-11-30 02:26:31 -02:00
bbffa59497
Some changes to Scrapyd:
...
* Always start one process per spider
* Added max_proc_per_cpu option (defaults to 4)
* Return the number of spiders (instead of a list of them) in schedule.json
2010-11-29 17:19:05 -02:00
2557777c39
Updated doc referring to HTTP cache middleware
2010-11-24 13:27:44 -02:00
426b6fa100
docs/intro/install.rst: added -U flag to easy_install command
2010-11-22 13:50:19 -02:00
91a7c25797
* Made Response.meta attribute map to Request.meta attribute. Closes #290
...
* Record redirected URLs in redirect middleware. Closes #291
2010-11-18 12:51:54 -02:00
ac007802d6
Simplified installation guide, including lxml as alternative dependency to libxml2. Closes #280
2010-11-17 21:32:23 -02:00
5a5364d0c1
Updated documentation to point out that simplejson is now required if using Python 2.5, and to recommended switching to Python 2.6
2010-11-16 03:31:04 -02:00
d988ca1ec2
Some changes to scrapy deploy command:
...
* changed deploy section names to [deploy:target]
* project is now passed through a -p|--project option
* version can now be set in the target configuration
* switched meaning of -l and -L options
* updated documentation accordingly
2010-11-08 17:01:06 -02:00
0f69e7a191
Some changes to HTTP Cache middleware:
...
* made it use the project data storage by default (closes #279 )
* added HTTPCACHE_ENABLED setting (False by default) to enable it
* made HTTPCACHE_DIR = 'httpcache' by default (inside the project data storage)
* simplified HTTPCACHE_EXPIRATION_SECS semantics: zero means don't expire,
dropped support for negative numbers
* other minor doc improvements
2010-11-01 02:38:15 -02:00
3c94c6cb9b
fixed sphinx doc id
2010-11-01 02:31:20 -02:00
130276605b
Bind the web server and telnet server to a configurable interface (WEBSERVICE_HOST). The default is to bind to all interfaces. Also add documentation for WEBSERVICE_HOST and TELNETCONSOLE_HOST.
2010-11-01 00:59:04 -02:00
b76c5c597f
* Added support for project data storage ( closes #276 )
...
* Documented project file structure
* Moved default location of SQLite database to project data storage dir (closes #277 )
2010-10-31 03:25:37 -02:00
dfa6745e91
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-30 16:05:53 -02:00
a0d9b43031
fixed typo in scrapyd doc
2010-10-30 16:05:32 -02:00
d67152ab0f
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-30 01:56:12 -02:00
75451cbe84
scrapyd doc: fixed delversion.json example
2010-10-30 01:56:00 -02:00
a59bfb539d
* Added lxml backend for XPath selectors. Closes #147
...
* Added new setting (SELECTORS_BACKEND) to choose which backend to use
* Deprecated the extract_unquoted() function from selectors
* Made libxml2 optional by adding a dummy selector backend. Closes #260
--HG--
rename : scrapy/tests/test_selector.py => scrapy/tests/test_selector_libxml2.py
2010-10-25 14:47:10 -02:00
6c921896a5
Expanded documentation on deploy command and versions. Refs #261
2010-10-19 00:11:45 -02:00
1d567cdce6
Added new 'deploy' command. Closes #261
2010-10-18 22:38:46 -02:00
7d8f922df9
Added documentation for CLOSESPIDER_ERRORCOUNT setting. Refs #254
2010-10-18 22:36:30 -02:00
c96f17c43d
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-18 03:21:21 -02:00
98662e53ea
Formatting fix in Scrapyd doc
2010-10-17 03:20:23 -02:00
a3d85da96f
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-16 19:54:24 -02:00
5f65c26080
Some minor improvements to feature list in Scrapy at a Glance documentation page
2010-10-16 19:02:08 -02:00
d5c8caf07b
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-10 20:31:38 -02:00
b4fbc6c5fa
Updated Scrapy Tutorial to reference feed exports, instead a custom written pipeline, and extended item pipeline documentation to include a JSON writer.
2010-10-10 20:31:05 -02:00
aa4142e4ba
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-07 18:23:48 -02:00
f4accb6c7f
Updated dmoz xpaths of Scrapy tutorial
2010-10-07 18:22:01 -02:00
7826869cb2
Added missing colon
2010-09-28 16:44:53 -03:00
0bf9e4627c
added support to CloseSpider extension, for close the spider after N pages have been crawled. Using the CLOSESPIDER_PAGECOUNT setting. closes #253
2010-09-28 16:29:37 -03:00
279dcc245f
Fixed role name in Sphinx doc
2010-09-26 01:01:06 -03:00
9599bde3e9
Removed RequestLimitMiddleware
2010-09-22 16:09:13 -03:00
ed4aec187f
Ported code to use new unified access to spider settings, keeping backwards compatibility for old spider attributes. Refs #245
2010-09-22 16:09:13 -03:00
b6c2b55e5b
Splitted settings classes from settings singleton. Closes #244
...
--HG--
rename : scrapy/conf/__init__.py => scrapy/conf.py
rename : scrapy/conf/default_settings.py => scrapy/settings/default_settings.py
rename : scrapy/tests/test_conf.py => scrapy/tests/test_settings.py
2010-09-22 15:47:33 -03:00
2ebfa7e68d
Removed unneeded code (since autodoc is not used in Sphinx doc)
2010-09-22 10:52:02 -03:00
9288f622f9
Added formname parameter for FormRequest.from_response
2010-09-20 08:33:24 -03:00
bf467fc37a
Check 'dont_merge_cookies' membership in request.meta, instead of getting its value
2010-09-10 15:29:15 -03:00
7d14a52234
Reference dont_merge_cookies in list of special Request.meta keys
2010-09-09 21:54:26 -03:00
7f21a6384f
Documented handle_httpstatus_list request.meta key
2010-09-09 21:50:40 -03:00
f1c943543a
Added dont_retry request.meta key to make RetryMiddleware ignore requests. Closes #234
2010-09-09 21:43:44 -03:00
9f01e3e79e
Added dont_redirect request.meta key to make RedirectMiddleware ignore requests. Closes #233
2010-09-09 21:37:35 -03:00
7da79b90fe
Make url/body attributes of Request/Response objects read-only - use replace() to change them. Deprecation warning left for backwards compatibilty.
2010-09-08 00:15:11 -03:00
c1aab2f58e
Copy callback/errback attributes when copying Requests
2010-09-08 00:15:09 -03:00
e9ebebb230
Removed UrlFilterMiddleware from scrapy.contrib - see this snippet for an alternative: http://snippets.scrapy.org/snippets/12/
2010-09-07 17:51:02 -03:00
12b04b068f
make download_timeout configurable by request. closes #229
...
--HG--
extra : rebase_source : e57dfd4aeb98d48b04fc4d0c6469e9a85e4b33a8
2010-09-07 13:01:40 -03:00
9158e9d682
Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options
...
--HG--
rename : debian/scrapyd.cfg => debian/000-default
rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf
2010-09-07 09:17:25 -03:00
3414bf13ee
remove request_uploaded signal and move response_received and response_downloaded to downloader manager. closes #228
...
--HG--
extra : rebase_source : 4af0d2a01b34de8a21048bb7f4a66bfc484b3b8f
2010-09-06 23:23:14 -03:00
3c5ab10688
Added FAQ entry about __VIEWSTATE parameter
2010-09-06 13:17:08 -03:00
e3d67d74f7
docs/intro/overview.rst: add example of scraped data and introduce loaders
2010-09-06 10:04:00 -03:00
00d55fbbd1
Updated 'Scrapy at a glance' document replacing item pipeline example by a simpler usage of feed exports
2010-09-05 23:38:37 -03:00
766f2d910d
Renamed Request Handlers to Download Handlers
2010-09-05 19:35:53 -03:00
a5cf71cb06
Updated Ubuntu package signing key location
2010-09-05 19:04:15 -03:00
6bf52fb50e
Make telnet console and web service try a range of ports for binding, instead of just one. Closes #226
2010-09-05 06:48:08 -03:00
14e985b076
Updated Command line tool documentation
2010-09-05 05:29:58 -03:00
1190f97944
Updated settings documentation
2010-09-05 04:58:14 -03:00
ebdb733e95
Updated some old messages in Scrapy shell doc
2010-09-05 04:45:43 -03:00
2f12618890
Post reference to Scrapyd in FAQ
2010-09-05 04:35:27 -03:00
bf34094e5a
Added versionadded:: notice to new documentation topics
2010-09-04 03:30:45 -03:00
9f4b1e47a4
damn, really fix httpcache docs
2010-09-04 03:26:41 -03:00
7ad901640b
fix httpcache docs
2010-09-04 03:23:08 -03:00
1abaa79469
Make ignored schemes configurable in HttpCacheMiddleware. closes #224
...
--HG--
extra : rebase_source : 2e6e8b93c642290f9bd6eb634eb4c8cd6da07c75
2010-09-04 02:58:43 -03:00
7b9fa7fbaa
Don't filter out requests coming from spiders that don't define allowed_domains. Closes #225
2010-09-04 02:23:04 -03:00
37e9c5d78e
Added new Scrapy service with support for:
...
* multiple projects
* uploading scrapy projects as Python eggs
* scheduling spiders using a JSON API
Documentation is added along with the code.
Closes #218 .
--HG--
rename : debian/scrapy-service.default => debian/scrapyd.default
rename : debian/scrapy-service.dirs => debian/scrapyd.dirs
rename : debian/scrapy-service.install => debian/scrapyd.install
rename : debian/scrapy-service.lintian-overrides => debian/scrapyd.lintian-overrides
rename : debian/scrapy-service.postinst => debian/scrapyd.postinst
rename : debian/scrapy-service.postrm => debian/scrapyd.postrm
rename : debian/scrapy-service.upstart => debian/scrapyd.upstart
rename : extras/scrapy.tac => extras/scrapyd.tac
2010-09-03 15:54:42 -03:00
758d21b2f9
Simplified images pipeline by allowing it to be used without having to override it in your project. Closes #217
2010-08-31 16:03:08 -03:00
e7b3247a18
Updated some missing references to scrapy-ws script
2010-08-27 01:05:59 -03:00
e2ed27e4fd
Added documentation for Ubuntu packages. Refs #211
2010-08-23 21:28:32 -03:00
58b4cc2c32
Some minor fixes to contribution Contributing documentation
2010-08-23 00:23:14 -03:00
6585c1a28f
removed (somewhat hacky) MAIL_DEBUG setting
2010-08-22 22:42:00 -03:00
6da1162839
minor fixes to FAQ
2010-08-22 19:08:45 -03:00
b3753d34eb
Added FAQ entry about feed exports
2010-08-22 05:59:30 -03:00
cbfec4bb0e
Renamed webservice ManagerResource to CrawlerResource
...
--HG--
rename : scrapy/contrib/webservice/manager.py => scrapy/contrib/webservice/crawler.py
2010-08-22 05:48:03 -03:00
7546a0805c
Removed webservice Spiders and Extensions resources since they can now be accessed through the Execution Manager (aka. Crawler) resource
2010-08-22 05:38:46 -03:00
c1225e0f45
"parse" command refactoring. This fixes #173 and renders #106 invalid.
2010-08-22 05:04:17 -03:00
9fccc11363
Moved scrapy.extension.extensions singleton to a "extensions" attribute of the scrapy.project.crawler singleton. Refs #189
2010-08-22 02:15:11 -03:00
faf7a7da83
Moved scrapymanager singleton to scrapy.project module. Refs #189
...
Detail of changes:
* Moved scrapy.core.manager.ExecutionManager class to scrapy.crawler.Crawler
* Added scrapy.project.crawler singleton to reference a singleton instance of
Crawler class (previously known as scrapymanager)
* Left an alias scrapy.core.manager.scrapymanager to scrapy.project.crawler for
backwards compatibility (to be removed in Scrapy 0.11)
2010-08-22 02:10:53 -03:00
053d45e79f
Splitted stats collector classes from stats collection facility ( #204 )
...
* moved scrapy.stats.collector.__init__ module to scrapy.statscol
* moved scrapy.stats.collector.simpledb module to scrapy.contrib.statscol
* moved signals from scrapy.stats.signals to scrapy.signals
* moved scrapy/stats/__init__.py to scrapy/stats.py
* updated documentation and tests accordingly
--HG--
rename : scrapy/stats/collector/simpledb.py => scrapy/contrib/statscol.py
rename : scrapy/stats/__init__.py => scrapy/stats.py
rename : scrapy/stats/collector/__init__.py => scrapy/statscol.py
2010-08-22 01:24:07 -03:00
c276c48c91
Added settings to Scrapy shell variables
2010-08-21 05:10:06 -03:00
68f9fcffe8
genspider command refactoring. Also updated tests and doc
2010-08-21 04:46:48 -03:00
0da6132136
Made command-line too output more concise
2010-08-21 03:37:59 -03:00
50621b7ef3
Renamed command "start" to "runserver". Closes #209
...
--HG--
rename : scrapy/commands/start.py => scrapy/commands/runserver.py
2010-08-21 01:42:55 -03:00
9aefa242d5
Applied documentation patch provided by Lucian Ursu ( closes #207 )
2010-08-21 01:26:35 -03:00
f782245c5a
Removed obsolete files
2010-08-21 01:24:39 -03:00
1d3b9e2ca8
Scrapy shell refactoring
2010-08-20 11:26:14 -03:00
7858244dca
Scrapy shell: moved python console starting code to scrapy.utils.console and get rid of noisy console banners
2010-08-20 01:33:02 -03:00
30e2404d8f
updated FAQ entry to recommend using higher download delays
2010-08-19 17:59:52 -03:00
3d8151bb26
Added FAQ entry about response code 999
2010-08-19 16:51:51 -03:00
2ff5a83b7a
Added persistent execution queue (based on SQLite), and a new 'queue' command to control it. Closes #198
2010-08-19 02:55:52 -03:00
94ead94bf6
Improved documentation of Scrapy command-line tool
...
--HG--
rename : docs/topics/cmdline.rst => docs/topics/commands.rst
2010-08-19 00:04:52 -03:00
34554da201
Deprecated scrapy-ctl.py command in favour of simpler "scrapy" command. Closes #199 . Also updated documenation accordingly and added convenient scrapy.bat script for running from Windows.
...
--HG--
rename : debian/scrapy-ctl.1 => debian/scrapy.1
rename : docs/topics/scrapy-ctl.rst => docs/topics/cmdline.rst
2010-08-18 19:48:32 -03:00
a71521bfba
Default per-command settings are now specified in the default_settings attribute of the command object. Closes #201
2010-08-17 18:30:13 -03:00
ad3fd0afe8
fixed minor formatting issue with new feed exports doc
2010-08-17 14:37:59 -03:00
e741a807d2
Added new Feed exports extension with documentation and storage tests. Closes #197 .
...
Also deprecated File export pipeline (to be removed in Scrapy 0.11).
Still need to add tests for FeedExport main extension code.
2010-08-17 14:27:48 -03:00
3e3a66620b
Added support for returning deferreds from (some) signal handlers. Closes #193
2010-08-14 21:10:37 -03:00
1df2c17b78
updated old documentation references
2010-08-12 20:45:11 -03:00
43d47e5d9b
Some improvements to Item Pipeline ( closes #195 ):
...
* Made Item Pipeline Manager a subclass of scrapy.middleware.MiddlewareManager
* Added open_spider/close_spider methods with support for returning deferreds from them
* Inverted the process_item() arguments to be more friendly with deferred
callbacks (backwards compatibility kept through arguments introspection)
* Updated documentation with new methods and process_item() arguments change
2010-08-12 10:48:37 -03:00
9d38a99aa8
updated missing doc reference from previous commit
2010-08-10 17:47:04 -03:00
784722774b
moved scrapy.core.signals to scrapy.signals, keeping backwards compatibility
2010-08-10 17:40:53 -03:00
c359a34d7d
moved scrapy.core.exceptions to scrapy.exceptions, keeping backwards compatibility
...
--HG--
rename : scrapy/core/exceptions.py => scrapy/exceptions.py
2010-08-10 17:36:48 -03:00
b1c0280616
removed scheduler middleware doc, as scheduler middleware will be removed soon
2010-08-10 16:59:49 -03:00
c7d9f6e270
Added JSON item exporter with doc and unittests ( closes #192 ), and also:
...
* put all json exporters in scrapy.contrib.exporters and deprecated
scrapy.contrib.exporters.jsonlines to reduce module nesting
* use JSON exporter with EXPORT_FORMAT=json in file export pipeline
2010-08-07 15:52:59 -03:00
49851d7f55
Automated merge with http://hg.scrapy.org/scrapy-0.9
2010-08-02 17:20:55 -03:00
6c68e4ce15
fixed documentation typo
2010-08-02 17:20:13 -03:00
453e7bf38c
Scrapy logging refactoring ( closes #188 ):
...
* added Twisted log observer for Scrapy, with unittests
* use numeric values from Python logging module for log levels
* removed scrapy.log.exc() function - use scrapy.log.err() instead
* removed logmessage_received signal - write a (twisted) log observer instead
* dropped support for obsolete `domain` argument
* dropped support for old setting names: LOGLEVEL, LOGFILE (replaced by LOG_LEVEL, LOG_FILE)
* deprecated `component` argument
2010-08-02 08:49:14 -03:00
e145ec686c
Replaced default spider manager (TwistedPluginSpiderManger) with a simpler one that doesn't depend on Twisted Plugins infrastructure.
2010-07-30 17:30:32 -03:00
e2290a5359
Some changes to Crawl spider:
...
* added process_request attribute to rules
* removed docstrings, since it duplicates documentation
2010-07-22 18:40:35 -03:00
3e013f564b
update docs for defaultheaders middleware and change spider attribute to match global setting name
2010-07-16 16:17:08 -03:00
6883a99c1e
Automated merge with ssh://hg.scrapy.org/scrapy-0.9
2010-07-16 14:56:00 -03:00
b91d40ba78
Fixed grammar error in doc (patch by stav) - closes #176
2010-07-16 11:34:18 -03:00
90a04f0530
Automated merge with http://hg.scrapy.org/scrapy-0.9
2010-07-13 19:47:55 -03:00
9e37ec4230
fixed documentation typo ( closes #151 )
2010-07-13 19:03:02 -03:00
b3a65d3313
HTTPCACHE: Don't cache response with codes in HTTPCACHE_IGNORE_HTTP_CODES
2010-07-09 13:14:25 -03:00
2571e1b7aa
docs: Some DjangoItem docs improvements, closes #134 . Thanks tn!
2010-06-27 09:09:54 -03:00
115e9f2162
Added FAQ entry about running Scrapy deployment.
2010-06-14 18:21:12 -03:00
ede1df4b4f
updated copyright year, and indentation space
2010-06-14 07:16:51 -03:00
bd16d1cd48
Added SMTP-AUTH support to scrapy.mail ( closes #149 )
2010-06-13 17:14:46 -03:00
6a33d6c4d0
* Added Scrapy Web Service with documentation and tests.
...
* Marked Web Console as deprecated.
* Removed Web Console documentation to discourage its use.
2010-06-09 13:46:22 -03:00
73305b1eb3
Added support for Requests without callbacks ( #166 ) - the Spider.parse() method
...
is used in those cases.
Also removed Request.deferred attribute.
2010-06-08 18:18:02 -03:00
38b5793152
Some changes to telnet console:
...
* moved module from scrapy.management.telnet to scrapy.telnet (to minimize
nested modules)
* added signal for updating telnet console variables (fixes #165 )
--HG--
rename : scrapy/management/telnet.py => scrapy/telnet.py
2010-06-02 17:49:18 -03:00
031eb1e5ed
removed no longer used SpiderScheduler (obsoleted by ExecutionQueue)
2010-05-28 17:27:15 -03:00
a71dc295af
Some mail improvements and tests.
...
* Add mail_sent signal and use it in MailSender
* Add MAIL_DEBUG setting to not send mails when testing
* Add MailSender tests
2010-05-28 16:51:47 -03:00
6059221716
Compose: stop process on None value by default
...
By doing this, we can use str.lower as a processor safely without
checking whether the given value is None.
By passing stop_on_none=False as keyword argument, this behaviour can be changed.
Signed-off-by: Ping Yin <pkufranky@gmail.com>
2010-04-08 10:59:47 +08:00
15b879f845
ItemLoader: Update docs for {add,replace,get}_{value,xpath}
...
Signed-off-by: Ping Yin <pkufranky@gmail.com>
2010-05-18 17:54:25 +08:00
bfd9cb42e5
Automated merge with http://hg.scrapy.org/scrapy-0.8
2010-05-17 20:11:27 -03:00
076cdfd585
Added documentation about contributing to Scrapy
2010-05-17 20:10:46 -03:00
7a55158fed
fixed documentation bug (thanks rhill for reporting)
2010-05-11 11:25:03 -03:00
5d03405cac
FormRequest.from_response doc fix. closes #155
...
--HG--
extra : rebase_source : d54979f6a15e5e997072dcbbc6d43b426189312b
2010-04-26 22:28:07 -03:00
2121a30c74
added note about installing Zope.Interface in windows platforms
2010-04-24 18:19:52 -03:00
6c12106803
Remove shpinx warning introduced by shorter title overline
2010-04-18 23:42:56 -03:00
2f8c052484
#154 : Language fixes to the documentation
2010-04-18 23:39:54 -03:00
dfdac356af
added missing default values to file xporter doc
2010-04-02 02:49:18 -03:00
f19c939925
fixed doc typo
2010-03-26 08:28:32 -03:00
99a876754c
Improved "What else?" section of "Scrapy at a glance" overview
2010-03-20 20:24:18 -03:00
234fd709ad
fixed doc typo (thanks Victor)
2010-03-19 10:32:17 -03:00
184cf6684f
Remove HttpException references from docs. Since 0.7, scrapy returns non-200 as Response objects and does not raise HttpException anymore
2010-03-18 10:05:33 -03:00
17091902f3
Explicity say where to save item class in "Defining our item" section of tutorial
2010-03-12 14:12:49 -02:00
c925c9e9a0
Notify spider when requests are ignored by HttpErrorMiddleware, and generally when any call to process_spider_input raises an exception
2010-05-12 16:41:06 -03:00
c0d45846b8
Automated merge with ssh://hg.scrapy.org/scrapy-0.8
2010-04-26 22:29:45 -03:00
81f6502e37
Automated merge with http://hg.scrapy.org/scrapy-0.8/
2010-04-24 18:22:13 -03:00
658e6f15e9
Automated merge with ssh://hg.scrapy.org/scrapy-0.8
2010-04-18 23:44:59 -03:00
68a875edb0
update ENCODING_ALIASES setting default value in settings documentation topic
2010-04-07 10:54:54 -03:00
de32612c99
Automated merge with http://hg.scrapy.org/scrapy-0.8
2010-04-02 02:49:51 -03:00
db5c3df679
SEP12 implementation
...
* Rename BaseSpider.domain_name to BaseSpider.name
This patch implements the domain_name to name change in BaseSpider class and
change all spider instantiations to use the new attribute.
* Add allowed_domains to spider
This patch implements the merging of spider.domain_name and
spider.extra_domain_names in spider.allowed_domains for offsite checking
purposes.
Note that spider.domain_name is not touched by this patch, only not used.
* Remove spider.domain_name references from scrapy.stats
* Rename domain_stats to spider_stats in MemoryStatsCollector
* Use ``spider`` instead of ``domain`` in SimpledbStatsCollector
* Rename domain_stats_history table to spider_data_history and rename domain
field to spider in MysqlStatsCollector
* Refactor genspider command
The new signature for genspider is: genspider [options] <domain_name>.
Genspider uses domain_name for spider name and for the module name.
* Remove spider.domain_name references
* Update crawl command signature <spider|url>
* docs: updated references to domain_name
* examples/experimental: use spider.name
* genspider: require <name> <domain>
* spidermanager: renamed crawl_domain to crawl_spider_name
* spiderctl: updated references of *domain* to spider
* added backward compatiblity with legacy spider's attributes
'domain_name' and 'extra_domain_names'
2010-04-01 18:27:22 -03:00
2299deda66
updated wrong link in doc
2010-03-26 14:02:33 -03:00
7cf2f87e27
Automated merge with http://hg.scrapy.org/scrapy-0.8
2010-03-26 08:29:34 -03:00
1330697c3d
Some improvements to Response encoding support:
...
* added encoding aliases, configurable through a new ENCODING_ALIASES setting
* Response.encoding now returns the real encoding detected for the body
* simplified TextResponse API by removing body_encoding() and
headers_encoding() methods
* Response.encoding now tries to infer the encoding from the body always (it
was done before only on HtmlResponse and TextResponse)
* removed scrapy.utils.encoding.add_encoding_alias() function
* updated implementation of scrapy.utils.response function to reflect these API
changes
* updated documentation to reflect API changes
2010-03-25 15:47:10 -03:00
9ddcd1095d
sort setting alphabetically
2010-03-25 11:45:06 -03:00
4fa833c849
Added LOG_ENCODING setting
2010-03-24 12:13:38 -03:00
87e68e7438
Made MailSender non IO-blocking, and improved MailSender documentation
2010-03-22 13:37:37 -03:00
1dfc79b5d0
Automated merge with http://hg.scrapy.org/scrapy-0.8
2010-03-20 20:48:11 -03:00
264cd2e035
Automated merge with http://hg.scrapy.org/scrapy-0.8
2010-03-19 10:32:42 -03:00
d12cd22d5e
switched default scheduler order to DFO, which consumes less memory by default
2010-03-04 10:15:58 -02:00
180c091fb2
Fixed encoding issue (reported in #135 ) when the encoding declared in the HTTP header is unknown. This is the patch proposed by Rolando, with an update to the Request/Response documentation.
2010-02-24 14:01:29 -02:00
bbef0fe870
Automated merge with http://hg.scrapy.org/users/rolando/scrapy/
2010-02-20 11:12:37 -02:00
a3d22c7240
Automated merge with http://hg.scrapy.org/scrapy-0.8/
2010-02-19 23:11:24 -02:00
60961e5499
minor documentation fix (refs #135 )
2010-02-19 23:09:48 -02:00
c1f8198639
Added RANDOMIZE_DOWNLOAD_DELAY setting
2010-02-19 21:53:18 -02:00
a6a3f085a7
docs: added crawlspider v2 outline documentation
...
Sign-Off: Rolando Espinoza La fuente
2010-02-19 18:22:38 -04:00
7235040936
merged upstream
2010-02-19 17:41:45 -04:00
91f4d6dc51
docs: adds another spider example that yields multiples requests/items from a single callback
2010-02-18 16:51:05 -02:00
57d60eae39
sort settings doc alphabetically by setting name
2010-01-31 18:11:13 -02:00
67858af83c
fixed doc typo
2010-01-18 18:16:58 -02:00
08eeaf98a2
fixed description of LOG_STDOUT setting
2010-01-13 15:51:08 -02:00
48739ae60c
install.rst: added explanation about why libxml2 2.6.28 or above is required
2010-01-13 12:20:24 -02:00
1402da31c5
docs: fixed typos and updated code examples
2010-01-11 12:28:22 -04:00
d60412ce19
titlecased Scrapy easy_install and some fixes to sign_release.sh script
2009-12-13 14:23:31 -02:00
422d6facb2
Automated merge with http://hg.scrapy.org/scrapy-stable
2009-12-12 16:52:07 -02:00
9d50604d24
added |version| to documentation title
2009-12-12 16:51:59 -02:00
a953efd8e5
Automated merge with http://hg.scrapy.org/scrapy-stable
2009-12-12 15:40:16 -02:00
4ecc909bc1
Fix RobotsTxtMiddleware reference in doc
2009-12-04 15:37:24 -02:00
07344666e2
Move webconsole extensions doc to webconsole topic
2009-12-01 10:47:11 -02:00
e694c8ed02
Remove domain references in close spider extension doc
2009-11-30 11:38:56 -02:00
12a7ff7312
Rename Close domain to close spider in extensions doc
2009-11-30 11:36:18 -02:00
8d9cedd88b
Reorder signals doc to respect alphabetical order
2009-11-30 11:29:19 -02:00
93cc3d2715
Correct param formatting in item pipelines doc
2009-11-30 11:04:15 -02:00
6084be3b2e
added iter_all() function to scrapy.util.trackref module and improved memory leaks documentation. also added a new FAQ antry about memory issues
2009-11-28 16:21:59 -02:00
dd662e09d8
some minor fixes to scheduler middleware doc
2009-11-19 12:23:54 -02:00
f4e93700bd
Automated merge with http://hg.scrapy.org/scrapy-stable/
2009-11-19 10:44:02 -02:00
c4f77c4da0
minor fixes to images doc (thanks amccloud)
2009-11-16 11:15:25 -02:00
0d6aee1f12
updated wrong documentation
2009-11-13 20:03:56 -02:00
aeab5370cb
StatsCollector: ported methods to receive spider instances ( closes #113 ), removed list_domains() method, added iter_spider_stats() method
2009-11-14 20:28:59 -02:00
c4c6e7c8cd
Automated merge with http://hg.scrapy.org/scrapy-stable/
2009-11-13 20:04:39 -02:00
07655d05ea
renamed REQUESTS_PER_SPIDER setting to CONCURRENT_REQUESTS_PER_SPIDER
2009-11-13 14:38:22 -02:00
564abd10ad
Refactored HttpCache middleware:
...
* simplified code
* performance improvements
* removed awkward/unused domain sectorization
* it can now receive Settings on constructor
* added unittests
* added documentation about filesystem storage structure
Also made scrapy.conf.Settings objects instantiable with a dict which is used to override default settings.
2009-11-13 14:25:47 -02:00
db7fec1fef
fixed doc typo
2009-11-12 12:17:39 -02:00
415dec4e16
made offsite middleware log messages when filtering out requests
2009-11-12 10:17:21 -02:00
49e39bf1ba
fixed typo
2009-11-06 16:49:48 -02:00
791f4932dd
added clarification about versioning and api stability
2009-11-06 16:28:51 -02:00
74d0e82dbe
renamed CloseDomain extension to CloseSpider, and renamed CLOSEDOMAIN_* settings to CLOSESPIDER_*
...
--HG--
rename : scrapy/contrib/closedomain.py => scrapy/contrib/closespider.py
2009-11-06 15:54:17 -02:00
919cd5b789
renamed setting CONCURRENT_DOMAINS to CONCURRENT_SPIDERS
2009-11-06 15:44:11 -02:00
d604dca96d
renamed setting REQUESTS_PER_DOMAIN to REQUESTS_PER_SPIDER
2009-11-06 15:42:11 -02:00
7728a23e99
Changed item pipeline API to pass spider references (instead of domain names) to process_item() method
2009-11-06 13:46:36 -02:00
a432c1ee40
updated logging doc to include new spider argument in log functions
2009-11-04 14:49:24 -02:00
97c322707a
* Renamed domain_{opened,closed,idle} signals to spider_{opened,closed,idle}
...
* Changed them to pass spider instances only (no domains) (refs #105 )
2009-11-03 00:39:02 -02:00
904cde6513
added clarification about new dont_click argument of FormRequest.from_response() method
2009-10-29 13:47:10 -02:00
a244d23b89
added dont_click attr to FormRequest
2009-10-29 13:18:13 -02:00
b41c5b5d5b
fixed typo in intro/install doc (thanks phaithful)
2009-10-29 10:41:20 -02:00
9b5fef4f48
fixed typo in intro/install doc (thanks phaithful)
2009-10-28 09:34:31 -02:00
7296a7b889
added DEFAULT_RESPONSE_ENCODING setting
2009-10-21 16:13:41 -02:00
720bc166cf
updated new clickdata argument doc
2009-10-20 17:21:56 -02:00
6abb3c17ee
Improve FormRequest.from_response method to pass click data arguments to ClientForm library
2009-10-20 15:51:41 -02:00
2712d55cb9
Automated merge with http://hg.scrapy.org/scrapy-stable
2009-10-07 23:58:38 -02:00
bd481751d8
moved images pipeline documentation to stable doc
...
--HG--
rename : docs/experimental/images.rst => docs/topics/images.rst
2009-10-07 22:57:25 -02:00
b4d202a6b0
added note about memory usage extension not working on windows
2009-10-07 22:57:10 -02:00
937acd91d1
improved documentation of http proxy middleware
2009-10-07 21:00:34 -02:00
d03bf1f27f
Automated merge with http://hg.scrapy.org/scrapy-stable
2009-10-07 20:38:46 -02:00
37d9e015bb
minor fix to tutorial
2009-10-07 20:15:49 -02:00
bc64ca3e13
Add support to set http proxies per request, and obey enviroment variables http_proxy and no_proxy by default.
2009-10-05 04:10:22 -02:00
8aa7d153ae
rewrote of downloader handlers
...
* add REQUEST_HANDLERS setting with defaults for file, http and https schemes
* add documentation of new setting
* add unittests for all the builtin handlers
* remove unused getPage function
2009-10-05 04:10:22 -02:00
a0eec7eaf6
some typos fixes and updates to install doc
2009-09-29 09:44:02 -03:00
1646482bef
reformatted installation guide
2009-09-29 08:41:34 -03:00
4a9d9282bc
removed obsolete scrapy architecture dia diagram
2009-09-28 23:54:01 -03:00
5862ba7db7
modified doc to reflect the new spider callback return policy (lists not needed)
2009-09-22 11:25:40 -03:00
802f918b69
removed obsolete doc static file
2009-09-22 11:25:38 -03:00
6e93872955
updated installation guide for using releases
2009-09-17 11:06:55 -03:00
132557dd14
some deployment changes in preparation for the 0.7.0 release candidate
2009-09-16 22:40:36 -03:00
fd41f06056
added doc on how to enable an Item Pipeline component
2009-09-16 14:19:16 -03:00
404e7e09d7
changed spider doc references in BaseSpider class
2009-09-16 14:10:11 -03:00
062730cbd8
fix csv exporter documentation
2009-09-16 00:17:50 -03:00
56b292e057
XmlItemExporter: added built-in support for exporting multi-valued fields (for convenience)
2009-09-14 22:05:52 -03:00
e8960bf616
added runspider command to run spiders directly, without having to create a project
2009-09-14 22:05:14 -03:00
99467d4e6e
Changed (unstable) scheduler middleware API to receive spider (instead of domain) in enqueue_request method
2009-09-13 20:51:43 -03:00
921fc4f3bf
Big Scrapy core refactoring to pass around spider references instead of domains.
...
This is to avoid accessing the scrapy.spider.spiders singleton for "resolving"
spiders, which is considered an "evil" practice because it ties us to the
singleton model for the spider resolver, which is a bad thing.
This change will also work as the foundation for the API cleaning that we'll
perform for 0.8. We decided to introduce this change now to have a more common
basecode between 0.7 and 0.8, which will allow us to better support 0.7 until
0.8 is released.
However, this change doesn't modify the stable/documented API, nor does it
change the core logic. Those changes will land on the 0.8 branch, after 0.7 is
released.
--HG--
rename : scrapy/contrib/domainsch.py => scrapy/contrib/spiderscheduler.py
2009-09-12 14:34:18 -03:00
8d49dc2fb5
changed IMAGES_THUMBS setting to a dict instead of a list of tuples, and more improvements to images pipeline doc
2009-09-11 17:36:00 -03:00
e20f766792
fixed some typos
2009-09-11 16:55:37 -03:00
c2fe350f72
more changes to images pipeline doc
2009-09-11 16:53:36 -03:00
ada46a2dbb
styled imagesp doc
2009-09-11 15:30:46 -03:00
be0f2beef0
more cleanup to scheduler middelware doc, and permanentely moved to experimental doc
2009-09-11 13:27:31 -03:00
0af052b68f
removed confusing title
2009-09-11 12:19:18 -03:00
f3240748cb
changed link to scheduler middleware doc, now in experimental
2009-09-11 12:03:23 -03:00
3998a0cb58
added more scheduler middleware documentation, and moved it to experimental
...
--HG--
rename : docs/topics/scheduler-middleware.rst => docs/experimental/scheduler-middleware.rst
2009-09-11 11:58:53 -03:00
d242a20573
updated images pipeline doc
2009-09-11 11:47:12 -03:00
f1bb8dc2a3
first cleanup of spider manager api
...
- removed asdict() and reload() methods
- added list() method
- removed default spider
2009-09-10 19:06:46 -03:00
f85813cd94
added FAQ entry about scrapy recipes and community spiders
2009-09-10 18:32:50 -03:00
269724a2b7
added Debugger extension, removed StackTraceDump from extensions available by default
2009-09-08 22:32:17 -03:00
4ddfa9a2a3
stlyed downloaded middleware doc
2009-09-07 12:18:57 -03:00
e3df11e5bb
added module directive to spidermw documentation
2009-09-07 12:03:24 -03:00
827aa19c6e
removed obsolete scrapy.utils.db module
2009-09-04 17:38:14 -03:00
861a803cc3
removed obsolete RestrictMiddleware
2009-09-04 17:22:56 -03:00
7e2587169b
added missing middleware docs
2009-09-04 12:39:02 -03:00
aefb94063a
more updates to spider middleware doc
2009-09-04 13:46:04 -03:00
d04640be5c
some improvements to spider middleware doc
2009-09-04 13:29:16 -03:00
96bb223c13
removed (pretty useless) DebugMiddleware
2009-09-04 12:59:58 -03:00
8a715701ec
fixed another doc typo
2009-09-03 14:31:00 -03:00
3c1bb7bc40
fixed typo in djangoitems doc (thanks anibal)
2009-09-03 11:23:25 -03:00
0e7b2a6da5
write header line by default when using csv exporter
...
--HG--
extra : rebase_source : 2d2d7153dde5e3f77e682e16d2e4408f732f234e
2009-09-03 13:58:39 -03:00
596d2c4479
moved CoreStats extension to scrapy.contrib.corestats
...
--HG--
rename : scrapy/stats/corestats.py => scrapy/contrib/corestats.py
2009-09-01 23:00:49 -03:00
6a50af05d7
removed useless SpiderReloader extension
2009-09-01 22:49:15 -03:00
79851aefa6
moved SpiderProfiler extension to scrapy.contrib_exp and removed references from documentation
...
--HG--
rename : scrapy/contrib/spider/profiler.py => scrapy/contrib_exp/spiderprofiler.py
2009-09-01 22:38:37 -03:00
d3c51fd6f2
improved images pipeline documentation
2009-09-01 21:07:47 -03:00
18fd635124
another doc typo
2009-09-01 12:52:40 -03:00
538cc9803a
fixed doc typo
2009-09-01 12:47:53 -03:00
df0e1f005f
exporters doc: fixed example and some typos
2009-09-01 08:56:54 -03:00
ac8f46ce9e
added File Export Pipeline reference to Exporters doc
2009-08-31 21:01:35 -03:00
8d006e9ea1
moved item exporters doc to stable doc
...
--HG--
rename : docs/experimental/exporters.rst => docs/topics/exporters.rst
2009-08-31 20:47:12 -03:00
0b152c99b5
added File Export Pipeline, a wrapper to use Item Exporters as Item Pipelines
2009-08-31 20:40:41 -03:00
8fab524978
moved engine.getstatus() method to scrapy.utils.engine function, to leave reporting logic out of engine code. added est() shortcut to telnet console
2009-08-31 12:44:32 -03:00
884f0c878f
Stats collectin: fixed race condition between stats persistance and population of stats on domain close
2009-08-29 19:44:13 -03:00
895c70e036
doc: fixed some links to scrapy-ctl topic
2009-08-29 18:23:55 -03:00
64572124e0
added doc about SCRAPY_SETTINGS_MODULE
2009-08-29 18:20:13 -03:00
9dc7a3a373
some minor adjustments to logging doc
2009-08-29 18:10:28 -03:00
60cbf24c89
more cleanups to startproject and project templates
...
--HG--
rename : scrapy/templates/project/root/scrapy-ctl.py => scrapy/templates/project/scrapy-ctl.py
2009-08-29 04:29:47 -03:00
c87216f02b
doc: added missing :synopsis: to some modules
2009-08-29 03:46:58 -03:00
2f55d7932b
replaced :ref: by :doc: links in doc index
2009-08-29 03:37:59 -03:00
924219dd5a
- added reference documentation about scrapy-ctl.py script
...
- yet another refactor to cmdline module
- removed help command (use -h to help now)
--HG--
rename : docs/experimental/scripts.rst => docs/topics/scrapy-ctl.rst
2009-08-28 20:32:55 -03:00
609aed4553
added inspect_response() function for inspecting responses from spiders
2009-08-27 19:33:33 -03:00
ea4f16bae6
refactored scrapy shell implementation, dropping IPython dependency, and adding a new 'view' shortcut
2009-08-27 18:24:08 -03:00
fb39bca24f
added djangoitem doc
2009-08-26 08:44:22 -03:00
65ab611fb0
more updates to HttpErrorMiddleware doc
2009-08-26 08:30:18 -03:00
44783a3a06
minor improvements to FAQ entry
2009-08-26 00:18:58 -03:00
4dba75a73f
updated HttpErrorMiddleware doc
2009-08-25 20:13:58 -03:00
8a074c9cb5
removed scrapy-admin.py command, and left only scrapy-ctl as the only scrapy command
2009-08-24 15:43:36 -03:00
39540b188a
changed torrent in overview doc
2009-08-24 15:11:04 -03:00
4b5aa30867
minor update to tutorial
2009-08-24 14:34:17 -03:00
0363040884
doc: added FAQ entry about Accept-Language
2009-08-24 13:56:44 -03:00
85282a4b76
added scrapy commandline scripts doc
2009-08-24 12:02:44 -03:00
ead8cb9fac
removed documentation about ugly DontCloseDomain exception (which will be removed in the future)
2009-08-24 11:57:21 -03:00
74f706b356
renamed "parse_item" method of XMLFeedSpider to "parse_node", keeping backwards compatibility
2009-08-24 10:54:34 -03:00
31693eb90f
dropped "cache" attribute of Request and Response objects
2009-08-24 10:34:05 -03:00
0186c6937a
HTTP auth middleware: added doc and unittest
2009-08-24 08:07:20 -03:00
46aaea9f32
doc: improved documentation about debugging leaks
2009-08-23 20:36:00 -03:00
0ed849248f
some improvements to item exporters
...
- passed previous class attributes to instances attributes
- better handling of constructor arguments
- better coverage on unittets (including encoding)
- updated documentation with new changes
2009-08-23 05:48:35 -03:00
7461710ef1
added some missing dots
2009-08-21 21:54:10 -03:00
9635a7839c
rearranged documentation into a better organization
...
--HG--
rename : docs/topics/index.rst => docs/index.rst
2009-08-21 21:49:54 -03:00
4761e0c88d
minor doc correction
2009-08-21 19:11:59 -03:00
012fd1cc2f
moved api-stability.rst doc to root and updated it
...
--HG--
rename : docs/misc/api-stability.rst => docs/api-stability.rst
2009-08-21 16:29:23 -03:00
ef6f04eb06
moved doc about debugging memory leaks to its own topic and added doc about trackref module
2009-08-21 16:07:16 -03:00
4980eb4968
added titles to signals doc
2009-08-21 15:07:52 -03:00
daea3f8a0c
sphinx docs: replaced custom :exception: xref by standard :exc:
2009-08-21 15:05:06 -03:00
c08d3aa9cc
updated tutorial to use new items api
2009-08-21 14:16:27 -03:00
37760eb13f
improved consistency of logging settings to use LOG_*
2009-08-21 08:54:12 -03:00
f0689a9ee8
fixed error in link extractors doc, thanks tarasm
2009-08-21 08:34:53 -03:00
41f1972e47
removed unused TRACE log level and improved logging documentation
2009-08-20 18:17:48 -03:00
20e82335e2
rename some exporter methods and complete exporter tests refactoring
2009-08-20 14:09:24 -03:00
3152e66fec
deprecate domain_open signal and handle stats domain open/close directly from the engine
2009-08-20 10:54:58 -03:00
33b53c59d5
moved scrapy.xpath to scrapy.selector
...
--HG--
rename : scrapy/xpath/__init__.py => scrapy/selector/__init__.py
rename : scrapy/xpath/document.py => scrapy/selector/document.py
rename : scrapy/xpath/factories.py => scrapy/selector/factories.py
2009-08-19 21:50:52 -03:00
dcc90fc196
declared loaders api stable and updated example project to use them
...
--HG--
rename : docs/experimental/loaders.rst => docs/topics/loaders.rst
2009-08-19 21:39:59 -03:00
e8504a054c
moved scrapy.newitem to scrapy.item and declared newitem api officially stable. updated docs and example project. deprecated old ScrapedItem
2009-08-19 21:39:58 -03:00
314e8dea43
added new item exporter tests, introduced some api changes
2009-08-19 19:05:33 -03:00
907fad6da8
make sure input processors always receive iterables as input
2009-08-19 16:49:49 -03:00
741a6d7863
item exporters refactoring
...
--HG--
rename : scrapy/contrib/exporter/jsonexporter.py => scrapy/contrib/exporter/jsonlines.py
2009-08-19 13:09:39 -03:00
82e4b6adcf
merge with ismael repo
2009-08-18 15:38:20 -03:00
e01d31c498
another improvement to doc navbar
2009-08-18 15:35:53 -03:00
be4226cec5
merge
2009-08-18 15:21:39 -03:00
f88fb27851
fixed error in xpath selectors doc
2009-08-18 15:18:49 -03:00
67c0c6a9e4
corrected indentation in xpath selectors doc
2009-08-18 15:13:23 -03:00
ff837e5a45
doc: improved top navbar
2009-08-18 15:12:44 -03:00
428dfe0d4a
corrected the style of spiders documentation
2009-08-18 15:06:33 -03:00
0192282d07
reorganized doc and moved robotstxt doc inside downloader middlewares doc
2009-08-18 14:36:18 -03:00
33089d287d
merged topics and reference doc
2009-08-18 14:05:15 -03:00
7f30461410
added support for defining EXTENSIONS setting using dicts, like middleware settings
...
--HG--
rename : scrapy/tests/test_utils_middleware.py => scrapy/tests/test_utils_conf.py
rename : scrapy/utils/middleware.py => scrapy/utils/conf.py
2009-08-18 11:05:36 -03:00
1bbe7991dc
added documentation for ImagesPipeline
2009-08-18 09:35:32 -03:00
48b40bd620
renamed x method of selectors to select
2009-08-17 15:58:06 -03:00
59e0a83ad4
removed more obsolete adaptors code
2009-08-17 14:48:11 -03:00
8c2f62ba9c
added Item Exporters documentation
2009-08-14 09:16:29 -03:00
02b01c3d2b
loaders doc: fixed outdated line
2009-08-13 23:24:24 -03:00
6a74b51371
remove dupes words in loaders doc, and unused import in example
...
--HG--
extra : rebase_source : ea7886725af7a54bd1031cb28f92efbdfe921d9e
2009-08-13 23:23:08 -03:00
ecca7f6478
fixes to Item Loader doc
2009-08-13 13:30:50 -03:00
8131d674d4
fixed outdated documentation (refs #97 )
2009-08-13 09:24:41 -03:00
5aeab5b291
converted scrapy.item package to module
...
--HG--
rename : scrapy/item/models.py => scrapy/item.py
2009-08-12 21:31:50 -03:00
991afdf6b3
some minor fixes to loaders doc
2009-08-12 19:23:04 -03:00
e6dd4d0955
renamed ApplyConcat processor to MapCompose
2009-08-12 18:43:08 -03:00
4e74f32401
renamed Pipe processor to Compose and documented it
2009-08-12 18:09:36 -03:00
958c3d9940
fixed some links to item loaders doc
2009-08-12 17:42:01 -03:00
d94b633565
renamed ItemLoader method populate_item() to load_item()
2009-08-12 17:40:27 -03:00
1dc592882b
Moved Item Loader to its final location in scrapy.contrib.loader, and updated doc/tests
...
--HG--
rename : docs/experimental/itemparser.rst => docs/experimental/loaders.rst
rename : scrapy/contrib/itemparser/__init__.py => scrapy/contrib/loader/__init__.py
rename : scrapy/contrib/itemparser/common.py => scrapy/contrib/loader/common.py
rename : scrapy/contrib/itemparser/parsers.py => scrapy/contrib/loader/processor.py
rename : scrapy/tests/test_itemparser.py => scrapy/tests/test_contrib_loader.py
2009-08-12 16:49:07 -03:00
7cbbc3ffb0
Renamed Loader to ItemParser (SEP-8 proposal). Documentation and unittests also updated.
...
--HG--
rename : docs/experimental/loaders.rst => docs/experimental/itemparser.rst
rename : scrapy/newitem/loader/__init__.py => scrapy/contrib/itemparser/__init__.py
rename : scrapy/newitem/loader/expanders.py => scrapy/contrib/itemparser/parsers.py
rename : scrapy/tests/test_itemloader.py => scrapy/tests/test_itemparser.py
2009-08-12 16:49:05 -03:00
d4aa72d2bb
removed obsolete documentation about Robust Scraped Item and Adaptors
2009-08-12 13:50:15 -03:00
e69311ef38
added missing text to new stats collector methods
2009-08-11 16:37:52 -03:00
45cbe1333f
fix typo in stats docs
2009-08-11 16:30:04 -03:00
8f16e7f9fa
remove `default` parameter from max_value/min_value stats methods, update docs, and add new methods to dummy collector
2009-08-11 16:23:23 -03:00
cd8eed77c5
stats collector gains two new methods to store values only if greater/lower than previous value
2009-08-11 15:46:44 -03:00
3614998053
fixed error in doc
2009-08-11 09:23:45 -03:00
7ac7597950
removed obsolete scrapy.contrib.cluster
2009-08-10 21:02:46 -03:00
c0532712f9
removed unnecesary response ResponseSoup extension, and replaced by a utils function with cache support
2009-08-10 21:02:24 -03:00
d334c035c0
removed unnecesary ResponseLibxml2 extension and moved libxml2 document caching functionality to Libxml2Document using weak references
2009-08-10 20:52:43 -03:00
49bab4777b
XPathSelector: added 're' argument to add_xpath method, exposed selector attribute
2009-08-10 19:42:20 -03:00
50fcc67278
improved reducers examples
2009-08-10 10:13:30 -03:00
5be0cee577
added TreeExpander example
2009-08-09 20:54:22 -03:00
38f82e3993
loaders doc: added information about expanders/reducers declaration precendece, and other minor improvements
2009-08-09 18:06:12 -03:00
b296d4169e
minor doc update for making it more windows-friendly
2009-08-09 17:08:42 -03:00
91eea82eef
added XPathLoader for working with XPath Selectors more conveniently
2009-08-08 07:26:20 -03:00
db90e26a8b
renamed ItemLoader class to Loader
2009-08-07 14:39:30 -03:00
e585c6cac4
relocated experimental newitems/loaders doc, and added example for extending fields metadata
...
--HG--
rename : docs/experimental/newitem-loader.rst => docs/experimental/loaders.rst
rename : docs/experimental/newitem.rst => docs/experimental/newitems.rst
2009-08-07 14:28:58 -03:00
d95e99f585
Added documentation for Items and Loaders, removed obsolete Item Adaptors documentation
...
--HG--
rename : docs/experimental/topics/newitem/index.rst => docs/experimental/newitem.rst
2009-08-07 03:50:09 -03:00
32894643a0
added ListField documentation, ordered field reference alphabetically
2009-08-03 15:00:04 -03:00
c3427e075c
added domain_stats parameter to stats_domain_closed signal
2009-07-31 16:36:35 -03:00
73172b244d
added from_unicode_list() method to Field objects
2009-07-30 16:58:24 -03:00
8ec7c9e01c
WEBCONSOLE_PORT setting now defaults to 6080
2009-07-29 18:59:34 -03:00
fcc91901eb
finished cleaning up closedomain documentation, and updated default settings
2009-07-27 15:42:35 -03:00
09ba6927d7
Some changes to CloseDomain extension:
...
- added support for closing by item passed count (CLOSEDOMAIN_ITEMPASSED)
- removed support for sending notification emails (since that's the job of
another extension)
2009-07-27 15:23:50 -03:00
9da66698f3
moved httprepr() method (from Request and Response objects) to scrapy.utils functions
2009-07-25 18:56:12 -03:00
c615ace6bd
doc: updated google directory links in firebug guide
2009-07-24 13:14:36 -03:00
96c3cdbec2
improve OffsiteMiddleware reference docs
...
--HG--
extra : rebase_source : 3ed3f23fc1ec63b521ead029c5749898f3ab05d7
2009-07-24 12:59:38 -03:00
38c3f7d0b4
Some changes to logging of scraped items:
...
1. "Scraped Item" log level changed to DEBUG
2. "Dropped Item" log level changed to WARNING
3. added "Passed Item" log message with INFO level
2009-07-23 11:49:48 -03:00
e43e28bf1d
minimal doc improvement
2009-07-23 09:12:49 -03:00
6d24ae5920
added reference to working with relative xpaths in the tutorial
2009-07-23 09:05:14 -03:00
9baa6bb2a8
minor selectors doc fix
2009-07-23 01:56:56 -03:00
3c4afb23be
moved newitem from scrapy.contrib_exp to scrapy.newitem
2009-07-22 15:13:36 -03:00
202894dd8f
fixes to newitem doc
2009-07-22 10:25:22 -03:00
7d8ba0542c
fixed typo in doc
2009-07-21 17:38:46 -03:00
aa345e116a
Added spider middleware documentation
2009-07-21 17:19:19 -03:00
90f1d9e489
Added Scheduler middleware reference documentation
2009-07-21 16:52:27 -03:00
00d9ce6608
fixes to newitem doc
2009-07-21 12:22:49 -03:00
894a5d81d3
changed the newitem API to a dict-like interface
2009-07-21 12:20:49 -03:00
9867e7b2d6
removed references to obsolete ENABLED_SPIDERS_FILE setting
2009-07-21 01:11:48 -03:00
92b746d866
SimpledbStatsCollector: moved domain creation to constructor
2009-07-17 12:49:53 -03:00
292757f312
added link to architecture overview and fixed old link
2009-07-16 19:15:19 -03:00
136014d718
Added section about relative xpaths to XPathSelectors doc
2009-07-16 17:29:29 -03:00
0fd603e27e
Automated merge with http://hg.scrapy.org/users/ismael/scrapy-newitem/
2009-07-16 11:16:46 -03:00
c2f12327d6
added dumping of global scrapy stats at engine shutdown
2009-07-16 10:48:10 -03:00
a9069e8eaa
reorganized experimental doc in topics and ref, slitted long documents, introduced minor changes
2009-07-16 09:58:44 -03:00
1125825996
doc: minor updates to tutorial
2009-07-15 22:10:00 -03:00
5582305648
renamed old STATS_DEBUG setting to STATS_DUMP
2009-07-15 21:02:51 -03:00
7f281cf295
some minor updates to doc
2009-07-15 01:40:44 -03:00
8480f49cf6
Complete stats refactoring and documentation
...
--HG--
rename : scrapy/stats/statscollector.py => scrapy/stats/collector/__init__.py
2009-07-15 00:30:40 -03:00
0227c53b13
reorganize newitem documentation
2009-07-14 15:55:52 -03:00
7fb889349e
fixed link to experimental doc
2009-07-14 08:46:15 -03:00
f1dc4e4af2
corrected TextField examples
2009-07-14 08:39:38 -03:00
a8aae41a13
removed unused files
2009-07-13 22:16:43 -03:00
5c39d173a5
merged proposed and experimental documentation, as it didn't make sense to keep two separate sections
...
--HG--
rename : docs/proposed/_images/scrapy_architecture.odg => docs/experimental/_images/scrapy_architecture.odg
rename : docs/proposed/_images/scrapy_architecture.png => docs/experimental/_images/scrapy_architecture.png
rename : docs/proposed/index.rst => docs/experimental/index.rst
rename : docs/proposed/newitem-fields.rst => docs/experimental/newitem-fields.rst
rename : docs/proposed/newitem.rst => docs/experimental/newitem.rst
2009-07-13 22:15:54 -03:00
26bb8ef608
doc: improved newitem fields reference
2009-07-13 22:05:48 -03:00
2bf39b7cdb
minor layout cleanups to newitem doc
2009-07-13 22:05:18 -03:00
74fcfc2cfc
deprecated old adaptors documentation
2009-07-13 22:03:56 -03:00
47d937f36b
only accept unicode strings in text fields
2009-07-13 15:54:48 -03:00
d75afaa161
renamed StringField to TextField
2009-07-13 15:54:46 -03:00
dff510384b
doc: updated SCHEDULER_MIDDLEWARES_BASE setting
2009-07-13 13:33:47 -03:00
b44409a203
added TimeField to newitem
2009-07-13 10:31:32 -03:00
5054b67a02
improved newitems doc and marked robust scraped items as deprecated
2009-07-11 21:26:52 -03:00
d5d2c5c924
update documentation to recent pydispatcher import path change
2009-07-09 17:13:30 -03:00
18fbd7c7eb
Automated merge with ssh://hg.scrapy.org/scrapy
2009-07-09 16:58:07 -03:00
eff8ea6173
remove response from item_passed and item_dropped signal api
2009-07-09 16:57:03 -03:00
5da32d9f6d
fixed Sphinx warning
2009-07-09 16:50:13 -03:00
ae7333d598
added simplejson optional dependency to doc
2009-07-09 16:49:20 -03:00
60e7b80798
removed signal docs from core.signals module, to leave them only in once place (the doc)
2009-07-09 12:57:10 -03:00
9e3e41f946
added more newitem documentation in proposed
2009-07-09 11:29:04 -03:00
b071681cd4
removed duplicated spiders doc (which used autodoc)
2009-07-09 11:14:33 -03:00
4f19115a80
removed old setting from default_settings.py, updated doc of CONCURRENT_ITEMS setting
2009-07-09 10:56:15 -03:00
8b26e49636
Added new ItemProcessor component to Scraper component
2009-07-08 23:48:06 -03:00
0c4c153819
improved Scrapy documentation index for better usability
2009-07-01 09:51:57 -03:00
80cd534f92
removed redundant botname from log lines
2009-06-25 16:48:04 -03:00
834ac9fca0
Some telnet console changes:
...
- added telnet console documentation
- added documentation for debugging memory leaks with guppy
- sorted out shell alises
- set default port (TELNETCONSOLE_PORT) to 6023
2009-06-23 16:08:58 -03:00
a8fd107ad4
engine: some extra simplifications and removed debug mode
2009-06-22 23:01:41 -03:00
400a54bf7c
Added reasons when closing domains ('reason' argument to engine close_domain method), replaced old 'status' parameter by new 'reason' parameter in domain_closed signal. updated and improved signals doc
2009-06-21 22:00:16 -03:00
adccd9a04e
Sorted out Duplicate Filter API.
...
--HG--
rename : scrapy/dupefilter/__init__.py => scrapy/contrib/dupefilter.py
2009-06-20 20:29:07 -03:00
47970e91bc
core: Invert request priority meaning, a higher request.priority value means more priority
2009-06-20 19:23:26 -03:00
18b6fecc47
Remove custom redirection priority of request returned by downloadermiddleware
2009-06-20 19:19:07 -03:00
161335fe78
Added domain schedulers (whose functionality was previously mixed with the
...
Scrapy Scheduler) and removed domain prioritizers whose functionality became
duplicated by the new domain schedulers.
2009-06-19 17:55:54 -03:00
3cb12e5dd3
Moved init_domain functionality out of the engine (refs #88 ) and into the
...
spider level. A new spider (InitSpider, not yet documented) was added to
provide initialization facilities.
Also renamed make_request_from_url to make_requests_from_url and allowed it to
return iterables.
2009-06-18 14:43:56 -03:00
fd0e490157
added StatsMailer extension
2009-06-12 15:38:21 -03:00
4a1a01354b
Added 'priority' attribute to Requests and removed old 'priority' argument passed through engine, scheduler and scheduler middleware calls
2009-06-11 22:25:47 -03:00
635ac1ca64
Simplified domain prioritizers, so that they don't receive domains in the
...
constructor (domain prioritizers will be refactored later anyway) and
simplified Scrapy Manager code thanks to this.
Added make_request_from_url method to BaseSpider, splitting funtionality to
create requests from URLs which was previously done all in start_requests.
2009-06-10 14:21:36 -03:00
1aac694343
updated settings doc
2009-05-28 13:52:56 -03:00
04e7f8f5f6
merged with Daniel's HttpException-removal branch
2009-05-28 13:45:26 -03:00
7eaa3ed24d
stop raising HttpException at download handlers and adapt download middlewares
2009-05-27 16:51:36 -03:00
c8827552b6
fix typo at WEBCONSOLE_ENABLED setting documentaion of default value. thanks dzen
2009-05-26 15:48:34 -03:00
6d1ffa7137
renamed CrawlDebug downloader middleware to DebugMiddleware
2009-05-25 20:14:50 -03:00
b1dad251ae
Deprecated Common Downloader Middleware and added DefaultHeaders Downloader
...
Middleware
2009-05-25 14:41:06 -03:00
90d408b04f
Some changes to HTTP cache middleware:
...
* documented
* moved from scrapy.contrib.downloadermiddleware.cache.CacheMiddleware to
scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware
* settings prefix changed from CACHE2_ to HTTPCACHE_
--HG--
rename : scrapy/contrib/downloadermiddleware/cache.py => scrapy/contrib/downloadermiddleware/httpcache.py
2009-05-24 19:13:06 -03:00
befd28eef4
docs/tutorial: added reminder about adding pipeline to ITEM_PIPELINES settings (thanks jamie)
2009-05-20 00:57:44 -03:00
04610a25dc
fixed bug in tutorial regarding csv writer pipeline, and other minor corrections
2009-05-19 03:07:08 -03:00
abfc52cd17
docs: modify install document to mercurial based installation instructions
2009-05-19 01:50:44 -03:00
86498abdf1
Sorted out Link Extractors organization by moving all them to
...
scrapy.contrib.linkextractors.
The most relevant being:
scrapy.link.extractors.RegexLinkExtractor
which was moved to:
scrapy.contrib.linkextractors.sgml.SgmlLinkExtractor
The old location still works but throws a deprecation warning. It will be
removed before the 0.7 release.
Documentation and tests were also updated.
Also, in this changeset, a new regex-based link extractor was added to
scrapy.contrib.linkextractors.regex.
--HG--
rename : scrapy/tests/sample_data/link_extractor/regex_linkextractor.html => scrapy/tests/sample_data/link_extractor/sgml_linkextractor.html
rename : scrapy/tests/test_link.py => scrapy/tests/test_contrib_linkextractors.py
2009-05-18 19:19:37 -03:00
766c8a4ea8
fixed some doc typos reported by phaithful
2009-05-14 08:32:48 -03:00
314a1c2bc2
improved configuration of middlewares using dicts and orders ( closes #85 )
2009-05-11 01:40:40 -03:00
c1c7b2d6c6
Sorted exceptions reference alphabetically
...
--HG--
extra : rebase_source : 1c6a192a76fcc90103ea324f6baf4387ba65e14a
2009-05-07 14:52:32 -03:00
ba69c29e37
mv scrapy/trunk to root as part of svn2hg migration
...
--HG--
rename : scrapy/trunk/AUTHORS => AUTHORS
rename : scrapy/trunk/INSTALL => INSTALL
rename : scrapy/trunk/LICENSE => LICENSE
rename : scrapy/trunk/README => README
rename : scrapy/trunk/bin/runtests.sh => bin/runtests.sh
rename : scrapy/trunk/docs/Makefile => docs/Makefile
rename : scrapy/trunk/docs/README => docs/README
rename : scrapy/trunk/docs/_ext/scrapydocs.py => docs/_ext/scrapydocs.py
rename : scrapy/trunk/docs/_static/items_adaptors-sample1.html => docs/_static/items_adaptors-sample1.html
rename : scrapy/trunk/docs/_static/scrapydoc.css => docs/_static/scrapydoc.css
rename : scrapy/trunk/docs/_static/selectors-sample1.html => docs/_static/selectors-sample1.html
rename : scrapy/trunk/docs/conf.py => docs/conf.py
rename : scrapy/trunk/docs/faq.rst => docs/faq.rst
rename : scrapy/trunk/docs/index.rst => docs/index.rst
rename : scrapy/trunk/docs/intro/index.rst => docs/intro/index.rst
rename : scrapy/trunk/docs/intro/install.rst => docs/intro/install.rst
rename : scrapy/trunk/docs/intro/overview.rst => docs/intro/overview.rst
rename : scrapy/trunk/docs/intro/tutorial.rst => docs/intro/tutorial.rst
rename : scrapy/trunk/docs/media/scrapy-architecture.dia => docs/media/scrapy-architecture.dia
rename : scrapy/trunk/docs/misc/api-stability.rst => docs/misc/api-stability.rst
rename : scrapy/trunk/docs/misc/index.rst => docs/misc/index.rst
rename : scrapy/trunk/docs/proposed/_images/scrapy_architecture.odg => docs/proposed/_images/scrapy_architecture.odg
rename : scrapy/trunk/docs/proposed/_images/scrapy_architecture.png => docs/proposed/_images/scrapy_architecture.png
rename : scrapy/trunk/docs/proposed/index.rst => docs/proposed/index.rst
rename : scrapy/trunk/docs/proposed/introduction.rst => docs/proposed/introduction.rst
rename : scrapy/trunk/docs/proposed/newitem.rst => docs/proposed/newitem.rst
rename : scrapy/trunk/docs/proposed/spiders.rst => docs/proposed/spiders.rst
rename : scrapy/trunk/docs/ref/downloader-middleware.rst => docs/ref/downloader-middleware.rst
rename : scrapy/trunk/docs/ref/email.rst => docs/ref/email.rst
rename : scrapy/trunk/docs/ref/exceptions.rst => docs/ref/exceptions.rst
rename : scrapy/trunk/docs/ref/extension-manager.rst => docs/ref/extension-manager.rst
rename : scrapy/trunk/docs/ref/extensions.rst => docs/ref/extensions.rst
rename : scrapy/trunk/docs/ref/index.rst => docs/ref/index.rst
rename : scrapy/trunk/docs/ref/link-extractors.rst => docs/ref/link-extractors.rst
rename : scrapy/trunk/docs/ref/logging.rst => docs/ref/logging.rst
rename : scrapy/trunk/docs/ref/request-response.rst => docs/ref/request-response.rst
rename : scrapy/trunk/docs/ref/selectors.rst => docs/ref/selectors.rst
rename : scrapy/trunk/docs/ref/settings.rst => docs/ref/settings.rst
rename : scrapy/trunk/docs/ref/signals.rst => docs/ref/signals.rst
rename : scrapy/trunk/docs/ref/spiders.rst => docs/ref/spiders.rst
rename : scrapy/trunk/docs/topics/_images/adaptors_diagram.png => docs/topics/_images/adaptors_diagram.png
rename : scrapy/trunk/docs/topics/_images/adaptors_diagram.svg => docs/topics/_images/adaptors_diagram.svg
rename : scrapy/trunk/docs/topics/_images/firebug1.png => docs/topics/_images/firebug1.png
rename : scrapy/trunk/docs/topics/_images/firebug2.png => docs/topics/_images/firebug2.png
rename : scrapy/trunk/docs/topics/_images/firebug3.png => docs/topics/_images/firebug3.png
rename : scrapy/trunk/docs/topics/_images/scrapy_architecture.odg => docs/topics/_images/scrapy_architecture.odg
rename : scrapy/trunk/docs/topics/_images/scrapy_architecture.png => docs/topics/_images/scrapy_architecture.png
rename : scrapy/trunk/docs/topics/adaptors.rst => docs/topics/adaptors.rst
rename : scrapy/trunk/docs/topics/architecture.rst => docs/topics/architecture.rst
rename : scrapy/trunk/docs/topics/downloader-middleware.rst => docs/topics/downloader-middleware.rst
rename : scrapy/trunk/docs/topics/extensions.rst => docs/topics/extensions.rst
rename : scrapy/trunk/docs/topics/firebug.rst => docs/topics/firebug.rst
rename : scrapy/trunk/docs/topics/firefox.rst => docs/topics/firefox.rst
rename : scrapy/trunk/docs/topics/index.rst => docs/topics/index.rst
rename : scrapy/trunk/docs/topics/item-pipeline.rst => docs/topics/item-pipeline.rst
rename : scrapy/trunk/docs/topics/items.rst => docs/topics/items.rst
rename : scrapy/trunk/docs/topics/link-extractors.rst => docs/topics/link-extractors.rst
rename : scrapy/trunk/docs/topics/robotstxt.rst => docs/topics/robotstxt.rst
rename : scrapy/trunk/docs/topics/selectors.rst => docs/topics/selectors.rst
rename : scrapy/trunk/docs/topics/settings.rst => docs/topics/settings.rst
rename : scrapy/trunk/docs/topics/shell.rst => docs/topics/shell.rst
rename : scrapy/trunk/docs/topics/spider-middleware.rst => docs/topics/spider-middleware.rst
rename : scrapy/trunk/docs/topics/spiders.rst => docs/topics/spiders.rst
rename : scrapy/trunk/docs/topics/stats.rst => docs/topics/stats.rst
rename : scrapy/trunk/docs/topics/webconsole.rst => docs/topics/webconsole.rst
rename : scrapy/trunk/examples/experimental/googledir/googledir/__init__.py => examples/experimental/googledir/googledir/__init__.py
rename : scrapy/trunk/examples/experimental/googledir/googledir/items.py => examples/experimental/googledir/googledir/items.py
rename : scrapy/trunk/examples/experimental/googledir/googledir/pipelines.py => examples/experimental/googledir/googledir/pipelines.py
rename : scrapy/trunk/examples/experimental/googledir/googledir/settings.py => examples/experimental/googledir/googledir/settings.py
rename : scrapy/trunk/examples/experimental/googledir/googledir/spiders/__init__.py => examples/experimental/googledir/googledir/spiders/__init__.py
rename : scrapy/trunk/examples/experimental/googledir/googledir/spiders/google_directory.py => examples/experimental/googledir/googledir/spiders/google_directory.py
rename : scrapy/trunk/examples/experimental/googledir/googledir/templates/spider_basic.tmpl => examples/experimental/googledir/googledir/templates/spider_basic.tmpl
rename : scrapy/trunk/examples/experimental/googledir/googledir/templates/spider_crawl.tmpl => examples/experimental/googledir/googledir/templates/spider_crawl.tmpl
rename : scrapy/trunk/examples/experimental/googledir/googledir/templates/spider_csvfeed.tmpl => examples/experimental/googledir/googledir/templates/spider_csvfeed.tmpl
rename : scrapy/trunk/examples/experimental/googledir/googledir/templates/spider_xmlfeed.tmpl => examples/experimental/googledir/googledir/templates/spider_xmlfeed.tmpl
rename : scrapy/trunk/examples/experimental/googledir/scrapy-ctl.py => examples/experimental/googledir/scrapy-ctl.py
rename : scrapy/trunk/examples/googledir/googledir/__init__.py => examples/googledir/googledir/__init__.py
rename : scrapy/trunk/examples/googledir/googledir/items.py => examples/googledir/googledir/items.py
rename : scrapy/trunk/examples/googledir/googledir/pipelines.py => examples/googledir/googledir/pipelines.py
rename : scrapy/trunk/examples/googledir/googledir/settings.py => examples/googledir/googledir/settings.py
rename : scrapy/trunk/examples/googledir/googledir/spiders/__init__.py => examples/googledir/googledir/spiders/__init__.py
rename : scrapy/trunk/examples/googledir/googledir/spiders/google_directory.py => examples/googledir/googledir/spiders/google_directory.py
rename : scrapy/trunk/examples/googledir/scrapy-ctl.py => examples/googledir/scrapy-ctl.py
rename : scrapy/trunk/extras/sql/scraping.sql => extras/sql/scraping.sql
rename : scrapy/trunk/profiling/priorityqueue/pq_classes.py => profiling/priorityqueue/pq_classes.py
rename : scrapy/trunk/profiling/priorityqueue/run.py => profiling/priorityqueue/run.py
rename : scrapy/trunk/profiling/priorityqueue/test_cases.py => profiling/priorityqueue/test_cases.py
rename : scrapy/trunk/scrapy/__init__.py => scrapy/__init__.py
rename : scrapy/trunk/scrapy/bin/scrapy-admin.py => scrapy/bin/scrapy-admin.py
rename : scrapy/trunk/scrapy/command/__init__.py => scrapy/command/__init__.py
rename : scrapy/trunk/scrapy/command/cmdline.py => scrapy/command/cmdline.py
rename : scrapy/trunk/scrapy/command/commands/__init__.py => scrapy/command/commands/__init__.py
rename : scrapy/trunk/scrapy/command/commands/crawl.py => scrapy/command/commands/crawl.py
rename : scrapy/trunk/scrapy/command/commands/download.py => scrapy/command/commands/download.py
rename : scrapy/trunk/scrapy/command/commands/genspider.py => scrapy/command/commands/genspider.py
rename : scrapy/trunk/scrapy/command/commands/help.py => scrapy/command/commands/help.py
rename : scrapy/trunk/scrapy/command/commands/list.py => scrapy/command/commands/list.py
rename : scrapy/trunk/scrapy/command/commands/log.py => scrapy/command/commands/log.py
rename : scrapy/trunk/scrapy/command/commands/parse.py => scrapy/command/commands/parse.py
rename : scrapy/trunk/scrapy/command/commands/shell.py => scrapy/command/commands/shell.py
rename : scrapy/trunk/scrapy/command/commands/start.py => scrapy/command/commands/start.py
rename : scrapy/trunk/scrapy/command/commands/stats.py => scrapy/command/commands/stats.py
rename : scrapy/trunk/scrapy/command/models.py => scrapy/command/models.py
rename : scrapy/trunk/scrapy/conf/__init__.py => scrapy/conf/__init__.py
rename : scrapy/trunk/scrapy/conf/commands/__init__.py => scrapy/conf/commands/__init__.py
rename : scrapy/trunk/scrapy/conf/commands/crawl.py => scrapy/conf/commands/crawl.py
rename : scrapy/trunk/scrapy/conf/commands/help.py => scrapy/conf/commands/help.py
rename : scrapy/trunk/scrapy/conf/commands/list.py => scrapy/conf/commands/list.py
rename : scrapy/trunk/scrapy/conf/commands/log.py => scrapy/conf/commands/log.py
rename : scrapy/trunk/scrapy/conf/commands/scrape.py => scrapy/conf/commands/scrape.py
rename : scrapy/trunk/scrapy/conf/commands/shell.py => scrapy/conf/commands/shell.py
rename : scrapy/trunk/scrapy/conf/commands/stats.py => scrapy/conf/commands/stats.py
rename : scrapy/trunk/scrapy/conf/commands/test.py => scrapy/conf/commands/test.py
rename : scrapy/trunk/scrapy/conf/default_settings.py => scrapy/conf/default_settings.py
rename : scrapy/trunk/scrapy/contrib/__init__.py => scrapy/contrib/__init__.py
rename : scrapy/trunk/scrapy/contrib/aws.py => scrapy/contrib/aws.py
rename : scrapy/trunk/scrapy/contrib/closedomain.py => scrapy/contrib/closedomain.py
rename : scrapy/trunk/scrapy/contrib/cluster/__init__.py => scrapy/contrib/cluster/__init__.py
rename : scrapy/trunk/scrapy/contrib/cluster/crawler/__init__.py => scrapy/contrib/cluster/crawler/__init__.py
rename : scrapy/trunk/scrapy/contrib/cluster/crawler/manager.py => scrapy/contrib/cluster/crawler/manager.py
rename : scrapy/trunk/scrapy/contrib/cluster/hooks/__init__.py => scrapy/contrib/cluster/hooks/__init__.py
rename : scrapy/trunk/scrapy/contrib/cluster/hooks/svn.py => scrapy/contrib/cluster/hooks/svn.py
rename : scrapy/trunk/scrapy/contrib/cluster/master/__init__.py => scrapy/contrib/cluster/master/__init__.py
rename : scrapy/trunk/scrapy/contrib/cluster/master/manager.py => scrapy/contrib/cluster/master/manager.py
rename : scrapy/trunk/scrapy/contrib/cluster/master/web.py => scrapy/contrib/cluster/master/web.py
rename : scrapy/trunk/scrapy/contrib/cluster/master/ws_api.txt => scrapy/contrib/cluster/master/ws_api.txt
rename : scrapy/trunk/scrapy/contrib/cluster/tools/scrapy-cluster-ctl.py => scrapy/contrib/cluster/tools/scrapy-cluster-ctl.py
rename : scrapy/trunk/scrapy/contrib/cluster/tools/test-worker.py => scrapy/contrib/cluster/tools/test-worker.py
rename : scrapy/trunk/scrapy/contrib/cluster/worker/__init__.py => scrapy/contrib/cluster/worker/__init__.py
rename : scrapy/trunk/scrapy/contrib/cluster/worker/manager.py => scrapy/contrib/cluster/worker/manager.py
rename : scrapy/trunk/scrapy/contrib/codecs/__init__.py => scrapy/contrib/codecs/__init__.py
rename : scrapy/trunk/scrapy/contrib/codecs/x_mac_roman.py => scrapy/contrib/codecs/x_mac_roman.py
rename : scrapy/trunk/scrapy/contrib/debug.py => scrapy/contrib/debug.py
rename : scrapy/trunk/scrapy/contrib/delayedclosedomain.py => scrapy/contrib/delayedclosedomain.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/__init__.py => scrapy/contrib/downloadermiddleware/__init__.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/cache.py => scrapy/contrib/downloadermiddleware/cache.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/common.py => scrapy/contrib/downloadermiddleware/common.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/cookies.py => scrapy/contrib/downloadermiddleware/cookies.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/debug.py => scrapy/contrib/downloadermiddleware/debug.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/errorpages.py => scrapy/contrib/downloadermiddleware/errorpages.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/httpauth.py => scrapy/contrib/downloadermiddleware/httpauth.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/httpcompression.py => scrapy/contrib/downloadermiddleware/httpcompression.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/redirect.py => scrapy/contrib/downloadermiddleware/redirect.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/retry.py => scrapy/contrib/downloadermiddleware/retry.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/robotstxt.py => scrapy/contrib/downloadermiddleware/robotstxt.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/stats.py => scrapy/contrib/downloadermiddleware/stats.py
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/useragent.py => scrapy/contrib/downloadermiddleware/useragent.py
rename : scrapy/trunk/scrapy/contrib/groupsettings.py => scrapy/contrib/groupsettings.py
rename : scrapy/trunk/scrapy/contrib/item/__init__.py => scrapy/contrib/item/__init__.py
rename : scrapy/trunk/scrapy/contrib/item/models.py => scrapy/contrib/item/models.py
rename : scrapy/trunk/scrapy/contrib/itemsampler.py => scrapy/contrib/itemsampler.py
rename : scrapy/trunk/scrapy/contrib/link_extractors.py => scrapy/contrib/link_extractors.py
rename : scrapy/trunk/scrapy/contrib/memdebug.py => scrapy/contrib/memdebug.py
rename : scrapy/trunk/scrapy/contrib/memusage.py => scrapy/contrib/memusage.py
rename : scrapy/trunk/scrapy/contrib/pipeline/__init__.py => scrapy/contrib/pipeline/__init__.py
rename : scrapy/trunk/scrapy/contrib/pipeline/images.py => scrapy/contrib/pipeline/images.py
rename : scrapy/trunk/scrapy/contrib/pipeline/media.py => scrapy/contrib/pipeline/media.py
rename : scrapy/trunk/scrapy/contrib/pipeline/s3images.py => scrapy/contrib/pipeline/s3images.py
rename : scrapy/trunk/scrapy/contrib/pipeline/show.py => scrapy/contrib/pipeline/show.py
rename : scrapy/trunk/scrapy/contrib/prioritizers.py => scrapy/contrib/prioritizers.py
rename : scrapy/trunk/scrapy/contrib/response/__init__.py => scrapy/contrib/response/__init__.py
rename : scrapy/trunk/scrapy/contrib/response/soup.py => scrapy/contrib/response/soup.py
rename : scrapy/trunk/scrapy/contrib/schedulermiddleware/__init__.py => scrapy/contrib/schedulermiddleware/__init__.py
rename : scrapy/trunk/scrapy/contrib/schedulermiddleware/duplicatesfilter.py => scrapy/contrib/schedulermiddleware/duplicatesfilter.py
rename : scrapy/trunk/scrapy/contrib/spider/__init__.py => scrapy/contrib/spider/__init__.py
rename : scrapy/trunk/scrapy/contrib/spider/profiler.py => scrapy/contrib/spider/profiler.py
rename : scrapy/trunk/scrapy/contrib/spider/reloader.py => scrapy/contrib/spider/reloader.py
rename : scrapy/trunk/scrapy/contrib/spidermiddleware/__init__.py => scrapy/contrib/spidermiddleware/__init__.py
rename : scrapy/trunk/scrapy/contrib/spidermiddleware/depth.py => scrapy/contrib/spidermiddleware/depth.py
rename : scrapy/trunk/scrapy/contrib/spidermiddleware/duplicatesfilter.py => scrapy/contrib/spidermiddleware/duplicatesfilter.py
rename : scrapy/trunk/scrapy/contrib/spidermiddleware/limit.py => scrapy/contrib/spidermiddleware/limit.py
rename : scrapy/trunk/scrapy/contrib/spidermiddleware/offsite.py => scrapy/contrib/spidermiddleware/offsite.py
rename : scrapy/trunk/scrapy/contrib/spidermiddleware/referer.py => scrapy/contrib/spidermiddleware/referer.py
rename : scrapy/trunk/scrapy/contrib/spidermiddleware/restrict.py => scrapy/contrib/spidermiddleware/restrict.py
rename : scrapy/trunk/scrapy/contrib/spidermiddleware/urlfilter.py => scrapy/contrib/spidermiddleware/urlfilter.py
rename : scrapy/trunk/scrapy/contrib/spidermiddleware/urllength.py => scrapy/contrib/spidermiddleware/urllength.py
rename : scrapy/trunk/scrapy/contrib/spiders/__init__.py => scrapy/contrib/spiders/__init__.py
rename : scrapy/trunk/scrapy/contrib/spiders/crawl.py => scrapy/contrib/spiders/crawl.py
rename : scrapy/trunk/scrapy/contrib/spiders/feed.py => scrapy/contrib/spiders/feed.py
rename : scrapy/trunk/scrapy/contrib/spiders/generic.py => scrapy/contrib/spiders/generic.py
rename : scrapy/trunk/scrapy/contrib/web/__init__.py => scrapy/contrib/web/__init__.py
rename : scrapy/trunk/scrapy/contrib/web/http.py => scrapy/contrib/web/http.py
rename : scrapy/trunk/scrapy/contrib/web/json.py => scrapy/contrib/web/json.py
rename : scrapy/trunk/scrapy/contrib/web/service.py => scrapy/contrib/web/service.py
rename : scrapy/trunk/scrapy/contrib/web/site.py => scrapy/contrib/web/site.py
rename : scrapy/trunk/scrapy/contrib/web/stats.py => scrapy/contrib/web/stats.py
rename : scrapy/trunk/scrapy/contrib/webconsole/__init__.py => scrapy/contrib/webconsole/__init__.py
rename : scrapy/trunk/scrapy/contrib/webconsole/enginestatus.py => scrapy/contrib/webconsole/enginestatus.py
rename : scrapy/trunk/scrapy/contrib/webconsole/livestats.py => scrapy/contrib/webconsole/livestats.py
rename : scrapy/trunk/scrapy/contrib/webconsole/scheduler.py => scrapy/contrib/webconsole/scheduler.py
rename : scrapy/trunk/scrapy/contrib/webconsole/spiderctl.py => scrapy/contrib/webconsole/spiderctl.py
rename : scrapy/trunk/scrapy/contrib/webconsole/spiderstats.py => scrapy/contrib/webconsole/spiderstats.py
rename : scrapy/trunk/scrapy/contrib/webconsole/stats.py => scrapy/contrib/webconsole/stats.py
rename : scrapy/trunk/scrapy/contrib_exp/__init__.py => scrapy/contrib_exp/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/adaptors/__init__.py => scrapy/contrib_exp/adaptors/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/adaptors/date.py => scrapy/contrib_exp/adaptors/date.py
rename : scrapy/trunk/scrapy/contrib_exp/adaptors/extraction.py => scrapy/contrib_exp/adaptors/extraction.py
rename : scrapy/trunk/scrapy/contrib_exp/adaptors/markup.py => scrapy/contrib_exp/adaptors/markup.py
rename : scrapy/trunk/scrapy/contrib_exp/adaptors/misc.py => scrapy/contrib_exp/adaptors/misc.py
rename : scrapy/trunk/scrapy/contrib_exp/downloadermiddleware/__init__.py => scrapy/contrib_exp/downloadermiddleware/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/downloadermiddleware/decompression.py => scrapy/contrib_exp/downloadermiddleware/decompression.py
rename : scrapy/trunk/scrapy/contrib_exp/history/__init__.py => scrapy/contrib_exp/history/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/history/history.py => scrapy/contrib_exp/history/history.py
rename : scrapy/trunk/scrapy/contrib_exp/history/middleware.py => scrapy/contrib_exp/history/middleware.py
rename : scrapy/trunk/scrapy/contrib_exp/history/scheduler.py => scrapy/contrib_exp/history/scheduler.py
rename : scrapy/trunk/scrapy/contrib_exp/history/store.py => scrapy/contrib_exp/history/store.py
rename : scrapy/trunk/scrapy/contrib_exp/link/__init__.py => scrapy/contrib_exp/link/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/newitem/__init__.py => scrapy/contrib_exp/newitem/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/newitem/adaptors.py => scrapy/contrib_exp/newitem/adaptors.py
rename : scrapy/trunk/scrapy/contrib_exp/newitem/fields.py => scrapy/contrib_exp/newitem/fields.py
rename : scrapy/trunk/scrapy/contrib_exp/newitem/models.py => scrapy/contrib_exp/newitem/models.py
rename : scrapy/trunk/scrapy/contrib_exp/pipeline/shoveitem.py => scrapy/contrib_exp/pipeline/shoveitem.py
rename : scrapy/trunk/scrapy/core/__init__.py => scrapy/core/__init__.py
rename : scrapy/trunk/scrapy/core/downloader/__init__.py => scrapy/core/downloader/__init__.py
rename : scrapy/trunk/scrapy/core/downloader/dnscache.py => scrapy/core/downloader/dnscache.py
rename : scrapy/trunk/scrapy/core/downloader/handlers.py => scrapy/core/downloader/handlers.py
rename : scrapy/trunk/scrapy/core/downloader/manager.py => scrapy/core/downloader/manager.py
rename : scrapy/trunk/scrapy/core/downloader/middleware.py => scrapy/core/downloader/middleware.py
rename : scrapy/trunk/scrapy/core/downloader/responsetypes/__init__.py => scrapy/core/downloader/responsetypes/__init__.py
rename : scrapy/trunk/scrapy/core/downloader/responsetypes/mime.types => scrapy/core/downloader/responsetypes/mime.types
rename : scrapy/trunk/scrapy/core/downloader/webclient.py => scrapy/core/downloader/webclient.py
rename : scrapy/trunk/scrapy/core/engine.py => scrapy/core/engine.py
rename : scrapy/trunk/scrapy/core/exceptions.py => scrapy/core/exceptions.py
rename : scrapy/trunk/scrapy/core/manager.py => scrapy/core/manager.py
rename : scrapy/trunk/scrapy/core/prioritizers.py => scrapy/core/prioritizers.py
rename : scrapy/trunk/scrapy/core/scheduler/__init__.py => scrapy/core/scheduler/__init__.py
rename : scrapy/trunk/scrapy/core/scheduler/middleware.py => scrapy/core/scheduler/middleware.py
rename : scrapy/trunk/scrapy/core/scheduler/schedulers.py => scrapy/core/scheduler/schedulers.py
rename : scrapy/trunk/scrapy/core/scheduler/store.py => scrapy/core/scheduler/store.py
rename : scrapy/trunk/scrapy/core/signals.py => scrapy/core/signals.py
rename : scrapy/trunk/scrapy/dupefilter/__init__.py => scrapy/dupefilter/__init__.py
rename : scrapy/trunk/scrapy/extension/__init__.py => scrapy/extension/__init__.py
rename : scrapy/trunk/scrapy/fetcher/__init__.py => scrapy/fetcher/__init__.py
rename : scrapy/trunk/scrapy/http/__init__.py => scrapy/http/__init__.py
rename : scrapy/trunk/scrapy/http/cookies.py => scrapy/http/cookies.py
rename : scrapy/trunk/scrapy/http/headers.py => scrapy/http/headers.py
rename : scrapy/trunk/scrapy/http/request/__init__.py => scrapy/http/request/__init__.py
rename : scrapy/trunk/scrapy/http/request/form.py => scrapy/http/request/form.py
rename : scrapy/trunk/scrapy/http/request/rpc.py => scrapy/http/request/rpc.py
rename : scrapy/trunk/scrapy/http/response/__init__.py => scrapy/http/response/__init__.py
rename : scrapy/trunk/scrapy/http/response/html.py => scrapy/http/response/html.py
rename : scrapy/trunk/scrapy/http/response/text.py => scrapy/http/response/text.py
rename : scrapy/trunk/scrapy/http/response/xml.py => scrapy/http/response/xml.py
rename : scrapy/trunk/scrapy/http/url.py => scrapy/http/url.py
rename : scrapy/trunk/scrapy/item/__init__.py => scrapy/item/__init__.py
rename : scrapy/trunk/scrapy/item/adaptors.py => scrapy/item/adaptors.py
rename : scrapy/trunk/scrapy/item/models.py => scrapy/item/models.py
rename : scrapy/trunk/scrapy/item/pipeline.py => scrapy/item/pipeline.py
rename : scrapy/trunk/scrapy/link/__init__.py => scrapy/link/__init__.py
rename : scrapy/trunk/scrapy/link/extractors.py => scrapy/link/extractors.py
rename : scrapy/trunk/scrapy/log/__init__.py => scrapy/log/__init__.py
rename : scrapy/trunk/scrapy/mail/__init__.py => scrapy/mail/__init__.py
rename : scrapy/trunk/scrapy/management/__init__.py => scrapy/management/__init__.py
rename : scrapy/trunk/scrapy/management/telnet.py => scrapy/management/telnet.py
rename : scrapy/trunk/scrapy/management/web.py => scrapy/management/web.py
rename : scrapy/trunk/scrapy/patches/__init__.py => scrapy/patches/__init__.py
rename : scrapy/trunk/scrapy/patches/monkeypatches.py => scrapy/patches/monkeypatches.py
rename : scrapy/trunk/scrapy/spider/__init__.py => scrapy/spider/__init__.py
rename : scrapy/trunk/scrapy/spider/manager.py => scrapy/spider/manager.py
rename : scrapy/trunk/scrapy/spider/middleware.py => scrapy/spider/middleware.py
rename : scrapy/trunk/scrapy/spider/models.py => scrapy/spider/models.py
rename : scrapy/trunk/scrapy/stats/__init__.py => scrapy/stats/__init__.py
rename : scrapy/trunk/scrapy/stats/corestats.py => scrapy/stats/corestats.py
rename : scrapy/trunk/scrapy/stats/statscollector.py => scrapy/stats/statscollector.py
rename : scrapy/trunk/scrapy/store/__init__.py => scrapy/store/__init__.py
rename : scrapy/trunk/scrapy/store/db.py => scrapy/store/db.py
rename : scrapy/trunk/scrapy/templates/project/module/__init__.py => scrapy/templates/project/module/__init__.py
rename : scrapy/trunk/scrapy/templates/project/module/items.py.tmpl => scrapy/templates/project/module/items.py.tmpl
rename : scrapy/trunk/scrapy/templates/project/module/pipelines.py.tmpl => scrapy/templates/project/module/pipelines.py.tmpl
rename : scrapy/trunk/scrapy/templates/project/module/settings.py.tmpl => scrapy/templates/project/module/settings.py.tmpl
rename : scrapy/trunk/scrapy/templates/project/module/spiders/__init__.py => scrapy/templates/project/module/spiders/__init__.py
rename : scrapy/trunk/scrapy/templates/project/module/templates/spider_basic.tmpl => scrapy/templates/project/module/templates/spider_basic.tmpl
rename : scrapy/trunk/scrapy/templates/project/module/templates/spider_crawl.tmpl => scrapy/templates/project/module/templates/spider_crawl.tmpl
rename : scrapy/trunk/scrapy/templates/project/module/templates/spider_csvfeed.tmpl => scrapy/templates/project/module/templates/spider_csvfeed.tmpl
rename : scrapy/trunk/scrapy/templates/project/module/templates/spider_xmlfeed.tmpl => scrapy/templates/project/module/templates/spider_xmlfeed.tmpl
rename : scrapy/trunk/scrapy/templates/project/root/scrapy-ctl.py => scrapy/templates/project/root/scrapy-ctl.py
rename : scrapy/trunk/scrapy/tests/__init__.py => scrapy/tests/__init__.py
rename : scrapy/trunk/scrapy/tests/run.py => scrapy/tests/run.py
rename : scrapy/trunk/scrapy/tests/sample_data/adaptors/enc-ascii.html => scrapy/tests/sample_data/adaptors/enc-ascii.html
rename : scrapy/trunk/scrapy/tests/sample_data/adaptors/enc-cp1252.html => scrapy/tests/sample_data/adaptors/enc-cp1252.html
rename : scrapy/trunk/scrapy/tests/sample_data/adaptors/enc-latin1.html => scrapy/tests/sample_data/adaptors/enc-latin1.html
rename : scrapy/trunk/scrapy/tests/sample_data/adaptors/enc-utf8-meta-latin1.html => scrapy/tests/sample_data/adaptors/enc-utf8-meta-latin1.html
rename : scrapy/trunk/scrapy/tests/sample_data/adaptors/enc-utf8.html => scrapy/tests/sample_data/adaptors/enc-utf8.html
rename : scrapy/trunk/scrapy/tests/sample_data/adaptors/extr_unquoted.xml => scrapy/tests/sample_data/adaptors/extr_unquoted.xml
rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.tar => scrapy/tests/sample_data/compressed/feed-sample1.tar
rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.xml => scrapy/tests/sample_data/compressed/feed-sample1.xml
rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.xml.bz2 => scrapy/tests/sample_data/compressed/feed-sample1.xml.bz2
rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.xml.gz => scrapy/tests/sample_data/compressed/feed-sample1.xml.gz
rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.zip => scrapy/tests/sample_data/compressed/feed-sample1.zip
rename : scrapy/trunk/scrapy/tests/sample_data/compressed/html-gzip.bin => scrapy/tests/sample_data/compressed/html-gzip.bin
rename : scrapy/trunk/scrapy/tests/sample_data/compressed/html-rawdeflate.bin => scrapy/tests/sample_data/compressed/html-rawdeflate.bin
rename : scrapy/trunk/scrapy/tests/sample_data/compressed/html-zlibdeflate.bin => scrapy/tests/sample_data/compressed/html-zlibdeflate.bin
rename : scrapy/trunk/scrapy/tests/sample_data/feeds/feed-sample1.xml => scrapy/tests/sample_data/feeds/feed-sample1.xml
rename : scrapy/trunk/scrapy/tests/sample_data/feeds/feed-sample2.xml => scrapy/tests/sample_data/feeds/feed-sample2.xml
rename : scrapy/trunk/scrapy/tests/sample_data/feeds/feed-sample3.csv => scrapy/tests/sample_data/feeds/feed-sample3.csv
rename : scrapy/trunk/scrapy/tests/sample_data/feeds/feed-sample4.csv => scrapy/tests/sample_data/feeds/feed-sample4.csv
rename : scrapy/trunk/scrapy/tests/sample_data/feeds/feed-sample5.csv => scrapy/tests/sample_data/feeds/feed-sample5.csv
rename : scrapy/trunk/scrapy/tests/sample_data/link_extractor/image_linkextractor.html => scrapy/tests/sample_data/link_extractor/image_linkextractor.html
rename : scrapy/trunk/scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html => scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html
rename : scrapy/trunk/scrapy/tests/sample_data/link_extractor/linkextractor_noenc.html => scrapy/tests/sample_data/link_extractor/linkextractor_noenc.html
rename : scrapy/trunk/scrapy/tests/sample_data/link_extractor/regex_linkextractor.html => scrapy/tests/sample_data/link_extractor/regex_linkextractor.html
rename : scrapy/trunk/scrapy/tests/sample_data/test_site/index.html => scrapy/tests/sample_data/test_site/index.html
rename : scrapy/trunk/scrapy/tests/sample_data/test_site/item1.html => scrapy/tests/sample_data/test_site/item1.html
rename : scrapy/trunk/scrapy/tests/sample_data/test_site/item2.html => scrapy/tests/sample_data/test_site/item2.html
rename : scrapy/trunk/scrapy/tests/test_adaptors.py => scrapy/tests/test_adaptors.py
rename : scrapy/trunk/scrapy/tests/test_aws.py => scrapy/tests/test_aws.py
rename : scrapy/trunk/scrapy/tests/test_c14nurls.py => scrapy/tests/test_c14nurls.py
rename : scrapy/trunk/scrapy/tests/test_contrib_response_soup.py => scrapy/tests/test_contrib_response_soup.py
rename : scrapy/trunk/scrapy/tests/test_dependencies.py => scrapy/tests/test_dependencies.py
rename : scrapy/trunk/scrapy/tests/test_downloadermiddleware_cookies.py => scrapy/tests/test_downloadermiddleware_cookies.py
rename : scrapy/trunk/scrapy/tests/test_downloadermiddleware_decompression.py => scrapy/tests/test_downloadermiddleware_decompression.py
rename : scrapy/trunk/scrapy/tests/test_downloadermiddleware_httpcompression.py => scrapy/tests/test_downloadermiddleware_httpcompression.py
rename : scrapy/trunk/scrapy/tests/test_downloadermiddleware_redirect.py => scrapy/tests/test_downloadermiddleware_redirect.py
rename : scrapy/trunk/scrapy/tests/test_downloadermiddleware_retry.py => scrapy/tests/test_downloadermiddleware_retry.py
rename : scrapy/trunk/scrapy/tests/test_downloadermiddleware_useragent.py => scrapy/tests/test_downloadermiddleware_useragent.py
rename : scrapy/trunk/scrapy/tests/test_dupefilter.py => scrapy/tests/test_dupefilter.py
rename : scrapy/trunk/scrapy/tests/test_engine.py => scrapy/tests/test_engine.py
rename : scrapy/trunk/scrapy/tests/test_http_cookies.py => scrapy/tests/test_http_cookies.py
rename : scrapy/trunk/scrapy/tests/test_http_headers.py => scrapy/tests/test_http_headers.py
rename : scrapy/trunk/scrapy/tests/test_http_request.py => scrapy/tests/test_http_request.py
rename : scrapy/trunk/scrapy/tests/test_http_response.py => scrapy/tests/test_http_response.py
rename : scrapy/trunk/scrapy/tests/test_http_url.py => scrapy/tests/test_http_url.py
rename : scrapy/trunk/scrapy/tests/test_item.py => scrapy/tests/test_item.py
rename : scrapy/trunk/scrapy/tests/test_itemadaptor.py => scrapy/tests/test_itemadaptor.py
rename : scrapy/trunk/scrapy/tests/test_libxml2.py => scrapy/tests/test_libxml2.py
rename : scrapy/trunk/scrapy/tests/test_link.py => scrapy/tests/test_link.py
rename : scrapy/trunk/scrapy/tests/test_newitem.py => scrapy/tests/test_newitem.py
rename : scrapy/trunk/scrapy/tests/test_pipeline_images.py => scrapy/tests/test_pipeline_images.py
rename : scrapy/trunk/scrapy/tests/test_responsetypes.py => scrapy/tests/test_responsetypes.py
rename : scrapy/trunk/scrapy/tests/test_robustscrapeditem.py => scrapy/tests/test_robustscrapeditem.py
rename : scrapy/trunk/scrapy/tests/test_schedulermiddleware_duplicatesfilter.py => scrapy/tests/test_schedulermiddleware_duplicatesfilter.py
rename : scrapy/trunk/scrapy/tests/test_serialization.py => scrapy/tests/test_serialization.py
rename : scrapy/trunk/scrapy/tests/test_spidermiddleware_duplicatesfilter.py => scrapy/tests/test_spidermiddleware_duplicatesfilter.py
rename : scrapy/trunk/scrapy/tests/test_spidermonkey.py => scrapy/tests/test_spidermonkey.py
rename : scrapy/trunk/scrapy/tests/test_spiders/__init__.py => scrapy/tests/test_spiders/__init__.py
rename : scrapy/trunk/scrapy/tests/test_spiders/testspider.py => scrapy/tests/test_spiders/testspider.py
rename : scrapy/trunk/scrapy/tests/test_stats.py => scrapy/tests/test_stats.py
rename : scrapy/trunk/scrapy/tests/test_storedb.py => scrapy/tests/test_storedb.py
rename : scrapy/trunk/scrapy/tests/test_utils_datatypes.py => scrapy/tests/test_utils_datatypes.py
rename : scrapy/trunk/scrapy/tests/test_utils_defer.py => scrapy/tests/test_utils_defer.py
rename : scrapy/trunk/scrapy/tests/test_utils_iterators.py => scrapy/tests/test_utils_iterators.py
rename : scrapy/trunk/scrapy/tests/test_utils_markup.py => scrapy/tests/test_utils_markup.py
rename : scrapy/trunk/scrapy/tests/test_utils_misc.py => scrapy/tests/test_utils_misc.py
rename : scrapy/trunk/scrapy/tests/test_utils_python.py => scrapy/tests/test_utils_python.py
rename : scrapy/trunk/scrapy/tests/test_utils_request.py => scrapy/tests/test_utils_request.py
rename : scrapy/trunk/scrapy/tests/test_utils_response.py => scrapy/tests/test_utils_response.py
rename : scrapy/trunk/scrapy/tests/test_utils_url.py => scrapy/tests/test_utils_url.py
rename : scrapy/trunk/scrapy/tests/test_webclient.py => scrapy/tests/test_webclient.py
rename : scrapy/trunk/scrapy/tests/test_xpath.py => scrapy/tests/test_xpath.py
rename : scrapy/trunk/scrapy/tests/test_xpath_extension.py => scrapy/tests/test_xpath_extension.py
rename : scrapy/trunk/scrapy/utils/__init__.py => scrapy/utils/__init__.py
rename : scrapy/trunk/scrapy/utils/c14n.py => scrapy/utils/c14n.py
rename : scrapy/trunk/scrapy/utils/datatypes.py => scrapy/utils/datatypes.py
rename : scrapy/trunk/scrapy/utils/db.py => scrapy/utils/db.py
rename : scrapy/trunk/scrapy/utils/defer.py => scrapy/utils/defer.py
rename : scrapy/trunk/scrapy/utils/display.py => scrapy/utils/display.py
rename : scrapy/trunk/scrapy/utils/http.py => scrapy/utils/http.py
rename : scrapy/trunk/scrapy/utils/iterators.py => scrapy/utils/iterators.py
rename : scrapy/trunk/scrapy/utils/markup.py => scrapy/utils/markup.py
rename : scrapy/trunk/scrapy/utils/misc.py => scrapy/utils/misc.py
rename : scrapy/trunk/scrapy/utils/python.py => scrapy/utils/python.py
rename : scrapy/trunk/scrapy/utils/request.py => scrapy/utils/request.py
rename : scrapy/trunk/scrapy/utils/response.py => scrapy/utils/response.py
rename : scrapy/trunk/scrapy/utils/serialization.py => scrapy/utils/serialization.py
rename : scrapy/trunk/scrapy/utils/test.py => scrapy/utils/test.py
rename : scrapy/trunk/scrapy/utils/url.py => scrapy/utils/url.py
rename : scrapy/trunk/scrapy/xlib/BeautifulSoup.py => scrapy/xlib/BeautifulSoup.py
rename : scrapy/trunk/scrapy/xlib/ClientForm.py => scrapy/xlib/ClientForm.py
rename : scrapy/trunk/scrapy/xlib/__init__.py => scrapy/xlib/__init__.py
rename : scrapy/trunk/scrapy/xlib/lrucache.py => scrapy/xlib/lrucache.py
rename : scrapy/trunk/scrapy/xlib/lsprofcalltree.py => scrapy/xlib/lsprofcalltree.py
rename : scrapy/trunk/scrapy/xlib/pydispatch/__init__.py => scrapy/xlib/pydispatch/__init__.py
rename : scrapy/trunk/scrapy/xlib/pydispatch/dispatcher.py => scrapy/xlib/pydispatch/dispatcher.py
rename : scrapy/trunk/scrapy/xlib/pydispatch/errors.py => scrapy/xlib/pydispatch/errors.py
rename : scrapy/trunk/scrapy/xlib/pydispatch/license.txt => scrapy/xlib/pydispatch/license.txt
rename : scrapy/trunk/scrapy/xlib/pydispatch/robust.py => scrapy/xlib/pydispatch/robust.py
rename : scrapy/trunk/scrapy/xlib/pydispatch/robustapply.py => scrapy/xlib/pydispatch/robustapply.py
rename : scrapy/trunk/scrapy/xlib/pydispatch/saferef.py => scrapy/xlib/pydispatch/saferef.py
rename : scrapy/trunk/scrapy/xlib/spidermonkey/INSTALL.scrapy => scrapy/xlib/spidermonkey/INSTALL.scrapy
rename : scrapy/trunk/scrapy/xlib/spidermonkey/__init__.py => scrapy/xlib/spidermonkey/__init__.py
rename : scrapy/trunk/scrapy/xlib/spidermonkey/sm_settings.py => scrapy/xlib/spidermonkey/sm_settings.py
rename : scrapy/trunk/scrapy/xlib/spidermonkey/spidermonkey.py => scrapy/xlib/spidermonkey/spidermonkey.py
rename : scrapy/trunk/scrapy/xpath/__init__.py => scrapy/xpath/__init__.py
rename : scrapy/trunk/scrapy/xpath/constructors.py => scrapy/xpath/constructors.py
rename : scrapy/trunk/scrapy/xpath/document.py => scrapy/xpath/document.py
rename : scrapy/trunk/scrapy/xpath/extension.py => scrapy/xpath/extension.py
rename : scrapy/trunk/scrapy/xpath/selector.py => scrapy/xpath/selector.py
rename : scrapy/trunk/scrapy/xpath/types.py => scrapy/xpath/types.py
rename : scrapy/trunk/scripts/rpm-install.sh => scripts/rpm-install.sh
rename : scrapy/trunk/setup.cfg => setup.cfg
rename : scrapy/trunk/setup.py => setup.py
2009-05-06 15:55:17 -03:00
Elias Dorneles
Ramiro Morales
Mikhail Korobov
Julia Medina
Pablo Hoffman
Faisal Anees
Mateusz Golewski
Daniel Graña
Shadab Zafar
Nicolás Alejandro Ramírez Quiros
Pablo Hoffman
Sudhanshu Shekhar
Jonas Tingeborn
Capi Etheriel
Artur Gaspar
Stefan
Lev Berman
tpeng
Jeff Paine
Lazar-T
HalfCrazy
John-Scott Atlakson
Mikael Åhlén
Uyounghz
Rocio Aramberri
Lisimia
Rolando Espinoza
Roy Crihfield
Zbigniew Siciarz
Paul Tremberth
Nikolaos-Digenis Karagiannis
Alexander Chekunkov
Rafal Jagoda
Rendaw
Carlos Rivera
ncp1113
Alexey Bezhan
Ana Sabina Uban
Paul Brown
Cash Costello
Breno Colom
Nikita Nikishin
tracicot
stray-leone
Hobson Lane
Hobson Lane
Ferdy Rodriguez
Hobson Lane
malcolm m
Yuri Prezument
RasPat1
Brian Lange
tntC4stl3
Travis Briggs
Dragon Dave
Loren Davie
irgmedeiros
cacovsky
Kumara Tharmalingam
Stefan
Berend Iwema
Hart
Rocio Aramberri
Dan
Juan M Uys
Steven Almeroth
Shane Evans
Jordi Llonch
whodatninja
Chris Tilden
Pedro Faustino
Emanuel Schorsch
Natan L
Hasnain Lakhani
Luan
Ilya Baryshev
Alex Cepoi
Artem Bogomyagkov
Valentin-Costel Hăloiu
Jason Yeo
lostsnow
Julien Duponchelle
dfdeshom
Martin Santos
Shuaib
Ismael Carnales
Ping Yin
Ping Yin
Lucian Ursu
Daniel Grana