RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 145 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
Daniel Grana 8724ee1674 utils: url canonicalization keeps blanks query parameters 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40960
 2009-03-03 15:05:50 +00:00
Ismael Carnales 2e1a170563 added LxmlParserLinkExtractor a new try on link extracting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40959
 2009-03-03 07:42:54 +00:00
Daniel Grana 309bf9277e xmlrpc: send request as POST by default and use dont_filter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40958
 2009-03-03 05:37:41 +00:00
Daniel Grana ff7c6d3782 http: add XmlRpcRequest based on xmlrpclib 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40957
 2009-03-03 00:37:06 +00:00
Daniel Grana ab3fdbdb8e core: prevent download timeout from been accidentally disabled 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40956
 2009-03-02 23:22:36 +00:00
Ismael Carnales 875e8e57d3 added lxml based link extractor for supporting sites with broken htm 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40955
 2009-03-02 19:49:42 +00:00
Ismael Carnales 127e5ae029 convert process_attr to a parameter in contructor so extending the class is not needed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40954
 2009-03-02 18:03:48 +00:00
Daniel Grana 1f38460ba4 allow returning None and non-iterables from spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40953
 2009-03-02 17:29:48 +00:00
Daniel Grana 2acdd9b409 utils: add any value to iterable function 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40952
 2009-03-02 17:29:18 +00:00
Ismael Carnales 315934bf6b added new LinkExtractor based on HTMLParser and with a hook for preprocessing the attribute 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40951
 2009-03-02 17:27:29 +00:00
Daniel Grana 57c7c5e9f6 images: log a short message when images request is ignored to dupefilter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40950
 2009-03-02 15:37:57 +00:00
Pablo Hoffman 4b0d69cc6d removed some unused imports 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40949
 2009-03-02 14:52:34 +00:00
Ismael Carnales d8f31f5678 pass the item_instance of ItemAdaptor as an adaptor_arg 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40948
 2009-03-02 02:58:10 +00:00
Ismael Carnales 16c20f5c4c added default_adaptor to ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40947
 2009-03-01 14:26:11 +00:00
Ismael Carnales f145e850ad discover adaptors in the python way 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40946
 2009-03-01 14:16:05 +00:00
Ismael Carnales 41ef6c262e added default_adaptor to the newitem documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40945
 2009-03-01 13:09:54 +00:00
Ismael Carnales 6724819423 updated newitem proposed docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40944
 2009-03-01 12:59:28 +00:00
Ismael Carnales ff950e7aa9 added intro/tutorial for new Item and ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40943
 2009-03-01 12:28:55 +00:00
Daniel Grana 43b87acd81 docs: rename setting doc accordly 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40942
 2009-03-01 11:14:30 +00:00
Daniel Grana e998f92297 docs: document SCHEDULER_MIDDLEWARE setting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40941
 2009-03-01 11:04:09 +00:00
Pablo Hoffman ee80b1bcc0 added doc about 'dont_merge_cookies' recently added by Damian 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40940
 2009-02-28 03:04:31 +00:00
Pablo Hoffman ca22e332d1 renamed to_list function to arg_to_list, added docstring and one more test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40939
 2009-02-28 02:51:18 +00:00
Pablo Hoffman 10be9b695d moved Ismael to Scrapy contributors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40938
 2009-02-28 02:42:46 +00:00
Pablo Hoffman 55fb3c0dd9 removed UsageError from docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40937
 2009-02-28 02:42:32 +00:00
Pablo Hoffman d16169e9ab replaced UsageError exception by more standard (and explicit) ones such as TypeError and ValueError 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40936
 2009-02-28 02:39:28 +00:00
Daniel Grana bfe5a554c2 newitem: clenaup ItemAdaptor inheritance search 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40935
 2009-02-27 18:48:09 +00:00
Pablo Hoffman e590764dc4 added Andres to AUTHORS 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40934
 2009-02-27 18:47:48 +00:00
Daniel Grana a88119aa8b newitem: redefine adaptor fields as staticmethods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40933
 2009-02-27 18:44:40 +00:00
samus_ 7215ffa1eb a bit of convention 
--HG--
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/http_compression.py => scrapy/trunk/scrapy/contrib/downloadermiddleware/httpcompression.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40932
 2009-02-25 09:53:44 +00:00
samus_ bf960fb102 renamed scrapy.contrib.downloadermiddleware.compression.CompressionMiddleware to scrapy.contrib.downloadermiddleware.http_compression.HTTPCompressionMiddleware 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40931
 2009-02-25 09:38:14 +00:00
Daniel Grana bfcde7e061 newitem: more tests on itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40930
 2009-02-25 09:31:39 +00:00
Daniel Grana fdaa2ec3ff newitem: support multiple depth inheritance of ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40929
 2009-02-25 09:15:28 +00:00
Daniel Grana 03db659040 newitem: make adaptor inheritance python2.5 compatible 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40928
 2009-02-25 08:44:58 +00:00
Ismael Carnales 0a598ff0b1 allow setting not adapted item values in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40927
 2009-02-25 08:28:24 +00:00
Ismael Carnales 6d2a3e093b added to_date adaptor to format a date suitable for DateField 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40926
 2009-02-25 07:42:35 +00:00
Daniel Grana 3e8945a477 newitem: another fix to ItemAdaptor inheritance and tests included 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40925
 2009-02-25 07:39:14 +00:00
Daniel Grana cf3afb9490 newitem: fix itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40924
 2009-02-25 06:52:06 +00:00
samus_ 1fa5c7bade back to the basics 8) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40923
 2009-02-25 06:48:41 +00:00
samus_ 5af4f3a279 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40922
 2009-02-25 06:37:17 +00:00
samus_ 2f60224b91 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40921
 2009-02-25 06:33:08 +00:00
Ismael Carnales 4675404fc1 ooops, removed extra print statement from adaptors.py 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40920
 2009-02-25 06:30:31 +00:00
Ismael Carnales 9fb0f5094e renamed extractors to adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40919
 2009-02-25 06:24:55 +00:00
samus_ c6e76dadc6 (missed parameter) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40918
 2009-02-25 05:49:23 +00:00
samus_ de1b9e5ec9 added gzip support for logs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40917
 2009-02-25 05:48:01 +00:00
Daniel Grana 82dc57e59b headers: complete new Headers behaviour migration. closes #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40916
 2009-02-25 00:29:10 +00:00
Daniel Grana 5de0816d9f url: add custom port based testcase 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40915
 2009-02-25 00:27:34 +00:00
Daniel Grana 5d3489ac10 headers: add docstring and title prior encoding keys. refs #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40914
 2009-02-25 00:27:02 +00:00
Pablo Hoffman 987ceb540e fixed import error (thanks Shane) and added some basic tests for ScrapedItem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40913
 2009-02-24 14:17:16 +00:00
Daniel Grana 934535740b headers: cleanup Headers class and add normvalue method to CaselessDict 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40912
 2009-02-24 06:58:12 +00:00
Daniel Grana f4b55bfd20 datatypes: add CaselessDict test and cleanup 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40911
 2009-02-24 06:57:28 +00:00