541451afb3
simplified obfuscated implementation of duplicated links removal and made it use scrapy.utils.python.unique function. also added missing 'unique' argument to docstring
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40620
2009-01-02 20:45:34 +00:00
c69d94fb45
removed unused private variable
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40619
2009-01-02 20:44:07 +00:00
f8a6e2fc7c
added key argument to unique function
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40618
2009-01-02 20:42:20 +00:00
e1c738a49e
minor text changes to home page
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40617
2009-01-02 20:17:40 +00:00
aa92755344
changed scrapy.org title to include more key words
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40616
2009-01-02 20:07:24 +00:00
371890008b
removed ugly apostrophe
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40615
2009-01-02 20:06:17 +00:00
fc36ad002f
renamed ItemPipeline class to ItemPipelineManager to avoid confusions
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40614
2009-01-02 19:59:41 +00:00
dd2f92f040
added comment about new docs
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40613
2009-01-02 19:55:22 +00:00
e95a28ef3d
added documentation for item pipeline, signals and exceptions. changed scrapy version to 0.7 in sphinx conf. closes #26 , #40 , #41
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40612
2009-01-02 19:48:31 +00:00
3789f064ff
added -E to sphinx-build so source rst's are be always reloaded and warning messages printed
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40611
2009-01-02 19:45:51 +00:00
47881c6e64
. Modified unicode conversion functions to accept the conversion encoding as a parameter
...
. Modified safe_url_string to make use of this change
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40610
2009-01-02 18:59:34 +00:00
63662ba6ff
. Added tests for RegexLinkExtractor
...
. Modified LinkExtractor in order to percent-encode urls using the response encoding
. Improved LinkExtractors processing of unique links
--HG--
rename : scrapy/trunk/scrapy/tests/sample_data/image_linkextractor.html => scrapy/trunk/scrapy/tests/sample_data/link_extractor/image_linkextractor.html
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40609
2009-01-02 18:32:55 +00:00
c791155aac
Modified url_safe_string in order to accept strings encoded in any encoding apart from unicode objects
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40608
2009-01-02 18:31:55 +00:00
95495f1d9d
removed domain_initialized signal which was defined but never used
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40607
2009-01-02 18:24:02 +00:00
12b56a3e39
removed border-bottom on link hover which made trac behave strangly
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40606
2009-01-02 17:15:22 +00:00
f7e4b91fe7
more rearrangements to scrapy doc
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40605
2009-01-02 16:34:10 +00:00
6a4a4a33ca
created test for body_or_str (forgot to add the file to the repo at r601)
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40604
2009-01-02 16:28:05 +00:00
a0f2ce0b53
rearranged doc according to what we agreed on a meeting with elpolilla and ismael
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40603
2009-01-02 16:25:28 +00:00
d706182e97
created test for body_or_str and made small improvement to scrapy.http.response
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40602
2009-01-02 16:21:28 +00:00
808923f5fd
added topics section to docs, changed reference to ref
...
--HG--
rename : scrapy/trunk/docs/reference/index.rst => scrapy/trunk/docs/ref/index.rst
rename : scrapy/trunk/docs/reference/settings.rst => scrapy/trunk/docs/ref/settings.rst
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40601
2009-01-02 16:08:18 +00:00
c1edf0e381
reverted generator approach because it conflicts with unique parameter plus fixed bug in canonicalize routine
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40600
2009-01-02 14:57:33 +00:00
e5a764cd2a
. Modified LinkExtractors extract_links for being inconsistent. Moved extra parameters to the constructors.
...
. Renamed ImageLinkExtractor to HtmlLinkExtractor and moved it to scrapy.contrib.link_extractors.
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40599
2009-01-02 14:06:51 +00:00
2979521b63
Reverted change in r590 for inconsistency in the addition operation (doesnt work equally in both ways)
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40598
2009-01-02 13:07:18 +00:00
ebd5f465aa
this changeset improves the extractors' implementation:
...
* moved LinkExtractor.extract_links to a private method and created wrapper in order to be able to work with text directly
* removed fugly new_response_from_xpaths from scrapy.utils.response and replaced it with a better internal algorithm
* moved former _normalize_input from scrapy.utils.iterators to scrapy.utils.response to fill the hole
* turned extractors' output lists into generators; this is safe because the result is always used in for..in constructs
* adapted test for generators (test should be rewritted anyways)
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40597
2009-01-02 08:42:36 +00:00
c9e48dc5ed
Disabled ImageLinkExtractor test for triggering mysterious leaks in libxml2
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40596
2009-01-02 04:12:59 +00:00
e0cb3c1ec4
Added test's missing sample file
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40595
2009-01-02 02:36:27 +00:00
91a23e61bf
Renamed LinkExtractors extract_urls method to extract_links
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40594
2009-01-02 02:34:44 +00:00
c82c799d07
Added ImageLinkExtractor's missing docstring
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40593
2009-01-02 02:25:17 +00:00
5757e54a28
- Added repr method to Link objects
...
- Added ImageLinkExtractor and tests
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40592
2009-01-02 02:18:11 +00:00
7bd73e0c09
Modified XPathSelectorLists: adding them should return a new XPathSelectorList
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40591
2008-12-31 13:23:25 +00:00
99143ff357
doc string fix
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40590
2008-12-30 19:38:15 +00:00
7e4c9200c2
corrected code blocks
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40589
2008-12-30 15:04:59 +00:00
1433af5561
fix to commit (r586)
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40588
2008-12-30 14:51:37 +00:00
710247e096
Allow to load a default spider when no spider was found for a given url. Also a generic spider
...
was added in contrib/spiders
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40587
2008-12-30 13:49:00 +00:00
d047409579
rearranged and sorted out default scrapy settings. closes #20
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40586
2008-12-30 13:34:57 +00:00
1f247b1efc
added settings documentation topic, and completed available settings reference. closes #30
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40585
2008-12-30 13:28:36 +00:00
af79bedf8f
Added doc line in RegexLinkExtractor for the case when no allow/deny argument is given
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40584
2008-12-30 13:04:06 +00:00
8ef8766070
the badge is back to green ... hulk is angry
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40583
2008-12-30 11:27:19 +00:00
ad2c2368a7
using small gray badge
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40582
2008-12-30 11:25:42 +00:00
c149e2de2c
removed menu from footer, added django badge
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40581
2008-12-30 11:16:32 +00:00
c9491b6041
replicating header menu in footer
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40580
2008-12-30 10:56:15 +00:00
81fecc0067
fixed settings indentation and added reference
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40579
2008-12-30 10:45:19 +00:00
36277f3a32
changed settings from description unit to crossreference
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40578
2008-12-29 19:02:56 +00:00
5a038e7d27
added docs breadrumb
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40577
2008-12-29 18:46:43 +00:00
e36f5624a7
fixed caps in news, reordered menu items
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40576
2008-12-29 18:35:13 +00:00
d619a11f88
make sidebar bigger
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40575
2008-12-29 18:28:40 +00:00
2f2c5cf19d
removed django.contrib.comments requirement
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40574
2008-12-29 18:27:16 +00:00
3d85932ae0
moved blog to news
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40573
2008-12-29 18:20:58 +00:00
48682189b4
using a modified version of django simple blog
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40572
2008-12-29 16:50:49 +00:00
a5b609ac5f
moved blog templates to backup folder
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40571
2008-12-29 16:13:19 +00:00
Pablo Hoffman
elpolilla
Ismael Carnales