RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,590 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

4590 Commits

Author SHA1 Message Date
Daniel Grana fb3a2dab2f core: allow returning None from spiders to support pipeline/download throttling 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40990
 2009-03-17 21:30:33 +00:00
Ismael Carnales c54a3fe7f6 removed item_to_dict 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40989
 2009-03-17 15:22:57 +00:00
Ismael Carnales 63150e2ed4 added item_to_dict util function 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40988
 2009-03-16 01:12:57 +00:00
Daniel Grana f9e5915fd4 newitem: change itemadaptor tests to avoid false positives 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40987
 2009-03-13 20:05:37 +00:00
Andres Moreira cbf6d76c5d Improved PriorityQueue and PriorityStack with a patch sent by Federico Feroldi. The new implementation is about two times faster than the old. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40986
 2009-03-12 19:06:57 +00:00
Ismael Carnales 518a7c87dd make field_adaptors non-public 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40985
 2009-03-11 13:26:29 +00:00
Ismael Carnales 2ba017b14d add tests for adaptation of multivaluated fields 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40984
 2009-03-11 13:01:26 +00:00
Daniel Grana 0be771152b stats: fix stats key typo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40983
 2009-03-11 03:01:30 +00:00
Pablo Hoffman f8ce241484 fixed MailSender documentation example - closes #74 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40982
 2009-03-10 19:56:22 +00:00
Ismael Carnales ec1a6af04a added MultiValuedField to newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40981
 2009-03-10 19:35:45 +00:00
Ismael Carnales 2f75c2c93e removed unneeded import 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40980
 2009-03-10 19:17:53 +00:00
Ismael Carnales 04baa7aed7 removed declarative from newitem (now using 'plain' metaclasses) added basic test for newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40979
 2009-03-10 18:40:26 +00:00
Ismael Carnales 774e7f623a fixed default adaptor inheritance, added test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40978
 2009-03-10 17:59:53 +00:00
Daniel Grana 67ab30a5ba core: add application/xhtml+xml support to responsetypes 
more info at http://www.w3.org/TR/xhtml-media-types/#media-types

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40977
 2009-03-10 17:51:03 +00:00
Ismael Carnales 63e6ce6e47 made IDENTITY a function of adaptor module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40976
 2009-03-10 16:57:27 +00:00
Ismael Carnales 698e6e73ea moved default_adaptor logic fully to the metaclass in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40975
 2009-03-10 16:34:48 +00:00
Ismael Carnales 06e049e551 move meta class initialization to metaclass in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40974
 2009-03-10 16:15:49 +00:00
Ismael Carnales f105359666 use metaclass __getattr__ to return default_adaptor in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40973
 2009-03-10 15:53:42 +00:00
Ismael Carnales 8bda81c01d add inheritance of item and adaptors to itemadaptor test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40972
 2009-03-10 14:03:27 +00:00
Ismael Carnales f9aa0a6a01 removed is_adaptor check in ItemAdaptor in favour of funcion adaptizer 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40971
 2009-03-10 13:38:32 +00:00
Pablo Hoffman 4093a4b83e added support to run scrapy tests with: python -m scrapy.tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40970
 2009-03-05 11:12:07 +00:00
Daniel Grana e3c9f5951a contrib: fix a header bug in httpcompression middleware and add tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40969
 2009-03-04 20:11:12 +00:00
Ismael Carnales 8ad4211e26 small changes to make code 3.0 compatible :) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40968
 2009-03-04 17:11:20 +00:00
Ismael Carnales ec22373b5a use ItemAdaptor.__classinit__ to set field_adaptors, also make defaulted field adaptors static_methods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40967
 2009-03-03 22:35:15 +00:00
Ismael Carnales b5b6b0cefc removed unneeded decorator from Item xD 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40966
 2009-03-03 21:32:37 +00:00
Ismael Carnales 154dded1de using declarative in ItemAdaptor, more to come 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40965
 2009-03-03 21:29:00 +00:00
Ismael Carnales e744e87517 code style changes for newitem, and only return fields or special attrs in Item 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40964
 2009-03-03 21:18:40 +00:00
Damian Canabal d26b968f51 fixing typo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40963
 2009-03-03 21:14:50 +00:00
Ismael Carnales 60de639434 added declarative module for declarative style classses and make Item use it 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40962
 2009-03-03 21:12:37 +00:00
Daniel Grana 0d47fdf6e8 utils: revert canonicalize_url implementation, it wasnt intended to be changed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40961
 2009-03-03 15:18:17 +00:00
Daniel Grana 8724ee1674 utils: url canonicalization keeps blanks query parameters 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40960
 2009-03-03 15:05:50 +00:00
Ismael Carnales 2e1a170563 added LxmlParserLinkExtractor a new try on link extracting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40959
 2009-03-03 07:42:54 +00:00
Daniel Grana 309bf9277e xmlrpc: send request as POST by default and use dont_filter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40958
 2009-03-03 05:37:41 +00:00
Daniel Grana ff7c6d3782 http: add XmlRpcRequest based on xmlrpclib 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40957
 2009-03-03 00:37:06 +00:00
Daniel Grana ab3fdbdb8e core: prevent download timeout from been accidentally disabled 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40956
 2009-03-02 23:22:36 +00:00
Ismael Carnales 875e8e57d3 added lxml based link extractor for supporting sites with broken htm 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40955
 2009-03-02 19:49:42 +00:00
Ismael Carnales 127e5ae029 convert process_attr to a parameter in contructor so extending the class is not needed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40954
 2009-03-02 18:03:48 +00:00
Daniel Grana 1f38460ba4 allow returning None and non-iterables from spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40953
 2009-03-02 17:29:48 +00:00
Daniel Grana 2acdd9b409 utils: add any value to iterable function 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40952
 2009-03-02 17:29:18 +00:00
Ismael Carnales 315934bf6b added new LinkExtractor based on HTMLParser and with a hook for preprocessing the attribute 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40951
 2009-03-02 17:27:29 +00:00
Daniel Grana 57c7c5e9f6 images: log a short message when images request is ignored to dupefilter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40950
 2009-03-02 15:37:57 +00:00
Pablo Hoffman 4b0d69cc6d removed some unused imports 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40949
 2009-03-02 14:52:34 +00:00
Ismael Carnales d8f31f5678 pass the item_instance of ItemAdaptor as an adaptor_arg 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40948
 2009-03-02 02:58:10 +00:00
Ismael Carnales 16c20f5c4c added default_adaptor to ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40947
 2009-03-01 14:26:11 +00:00
Ismael Carnales f145e850ad discover adaptors in the python way 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40946
 2009-03-01 14:16:05 +00:00
Ismael Carnales 41ef6c262e added default_adaptor to the newitem documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40945
 2009-03-01 13:09:54 +00:00
Ismael Carnales 6724819423 updated newitem proposed docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40944
 2009-03-01 12:59:28 +00:00
Ismael Carnales ff950e7aa9 added intro/tutorial for new Item and ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40943
 2009-03-01 12:28:55 +00:00
Daniel Grana 43b87acd81 docs: rename setting doc accordly 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40942
 2009-03-01 11:14:30 +00:00
Daniel Grana e998f92297 docs: document SCHEDULER_MIDDLEWARE setting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40941
 2009-03-01 11:04:09 +00:00