RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
461 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

461 Commits

Author SHA1 Message Date
Daniel Grana 7ac8ae993a Revert "add 505 and 403 to retry status codes due to amazon s3 random fails while uploading images" 
This reverts changeset r457

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40461
 2008-12-02 13:45:35 +00:00
elpolilla 068d2278a3 Removed silly bug from the previous revision 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40460
 2008-12-02 13:29:20 +00:00
elpolilla a4ba7b9d9a Added stripping of urls in LinkExtractor, and removed unnecesary check 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40459
 2008-12-02 13:26:04 +00:00
Daniel Grana c22223beb7 add 505 and 403 to retry status codes due to amazon s3 random fails while uploading images 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40458
 2008-12-02 13:01:49 +00:00
elpolilla 50379ee21e Added LinkExtractor.matches test and fixed bug in that same method 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40457
 2008-12-02 12:51:32 +00:00
elpolilla c75ac38b92 Improved parse command and LinkExtractor's matches method 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40456
 2008-12-02 12:20:55 +00:00
elpolilla eff09f2f78 Added some missing postprocessing to items in parse command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40455
 2008-12-02 02:17:15 +00:00
Daniel Grana 3b7f282bbd fix missing import errors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40454
 2008-12-01 18:44:04 +00:00
Daniel Grana dd37bb5a4c MONKEYPATCH: in twisted < 8.0.0, HTTPPageGetter class can not handle HEAD requests because of response empty body raising PartialDownloadError. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40453
 2008-12-01 17:14:06 +00:00
elpolilla aa3ea811d1 Fixed wrong error message in parse command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40452
 2008-12-01 15:49:04 +00:00
elpolilla 0cff67b25e Finished implementing new parse command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40451
 2008-12-01 15:44:50 +00:00
elpolilla 8b37282752 Changed CrawlSpider's way of setting crawling rules 
--HG--
rename : scrapy/trunk/scrapy/contrib/spiders2/crawl.py => scrapy/trunk/scrapy/contrib/spiders/crawl.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40450
 2008-12-01 02:59:34 +00:00
Daniel Grana 7f05b923ab s3 images pipeline: dont include path prefix in key name 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40449
 2008-11-28 16:18:03 +00:00
Pablo Hoffman 778b77f2bb added parse2 command (candidate to replace parse command) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40448
 2008-11-28 01:58:49 +00:00
Pablo Hoffman 6d5a130b96 more refactoring to CrawlSpider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40447
 2008-11-28 01:57:07 +00:00
elpolilla 1b92fef9a7 Added scrapy intro RST 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40446
 2008-11-27 15:44:29 +00:00
Pablo Hoffman 142cd72fae removed function check_valid_urlencode from scrapy.utils.url (it didn't fit there because it was too custom) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40445
 2008-11-27 15:21:41 +00:00
Ismael Carnales d865ca4e5b removed unused and unneeded SITE_URL from localsettings 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40444
 2008-11-27 14:54:42 +00:00
Ismael Carnales 2f5cff0b24 removed unneeded index view 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40443
 2008-11-27 14:53:02 +00:00
Ismael Carnales 9cc3b2881f updated requirements 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40442
 2008-11-27 14:46:01 +00:00
Ismael Carnales 7683eefc0d add empty docs/pickle dir for pickled docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40441
 2008-11-27 14:43:36 +00:00
Ismael Carnales dae37d3dff don't store pickled docs in SVN only the dir 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40440
 2008-11-27 14:42:42 +00:00
Ismael Carnales 9e43e69d13 added Makefile for building docs and testing 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40439
 2008-11-27 14:39:55 +00:00
elpolilla b714b5ab09 Added string cast in check_valid_urlencode to prevent urllib from failing 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40438
 2008-11-27 11:21:28 +00:00
Pablo Hoffman af842a3e6d added proper fix to canonicalize_url problem with unicode urls already percent-encoded 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40437
 2008-11-26 22:10:06 +00:00
samus_ 8d08ab5f98 added excluded characters, also implemented dan's readability tips 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40436
 2008-11-26 18:31:53 +00:00
Daniel Grana 874331a5b7 revert a testing less-than condition 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40435
 2008-11-26 18:28:49 +00:00
Pablo Hoffman ce172e8520 reverted wrong fix to canonicalize_url 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40434
 2008-11-26 17:55:03 +00:00
elpolilla 0958eecfca Modified canonicalize_url to make it always work with regular strings 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40433
 2008-11-26 16:18:19 +00:00
elpolilla 8eab5802cd Fixed bug in item.attribute 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40432
 2008-11-26 16:09:39 +00:00
elpolilla 40b3f06f0b Improved add parameter in item.attribute 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40431
 2008-11-26 14:48:07 +00:00
elpolilla 26b4be78e3 Minor correction in docstring 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40430
 2008-11-26 13:12:20 +00:00
elpolilla 04bd65de00 Added new crawling spider (with rules), but without replacing the previous one 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40429
 2008-11-26 12:49:14 +00:00
samus_ fe49bc2011 improved check 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40428
 2008-11-26 12:05:31 +00:00
Daniel Grana 537a3457cf s3: put image size as metadata 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40427
 2008-11-25 20:51:02 +00:00
samus_ 664a72490f improved url validation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40426
 2008-11-25 18:49:17 +00:00
Ismael Carnales 89bd6ede43 first commit of doc apps 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40425
 2008-11-25 18:16:04 +00:00
Ismael Carnales 4241171947 style h1s 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40424
 2008-11-25 17:42:10 +00:00
Daniel Grana c61d0648e6 use hashlib instead of deprecated md5 module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40423
 2008-11-25 17:17:25 +00:00
Daniel Grana 81416c4dd1 s3: add image md5 content checksum to image path 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40422
 2008-11-25 17:09:26 +00:00
Pablo Hoffman 33b20be7b5 cleanup handling of adaptor pipe debugging code 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40421
 2008-11-25 17:06:36 +00:00
Ismael Carnales e8df55d101 added header and footer templates 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40420
 2008-11-25 16:23:37 +00:00
elpolilla 011adf4dd7 Turned "strip_list" adaptor into "strip" 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40419
 2008-11-25 15:27:04 +00:00
Daniel Grana 1a60f6c1cb comment default thumb sizes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40418
 2008-11-25 13:44:06 +00:00
Daniel Grana f61be3f078 fix syntax typo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40417
 2008-11-25 11:58:31 +00:00
Daniel Grana 072fed8d4a aws: request signing function and tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40416
 2008-11-25 11:06:34 +00:00
Pablo Hoffman 0fbb7579f6 added new CrawlSpider that's gonna replace the old CrawlSpider soon 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40415
 2008-11-25 02:38:26 +00:00
elpolilla f558a49d2c Fixed ExtractImages adaptor, which wasnt looking for a base tag 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40414
 2008-11-25 01:41:22 +00:00
Pablo Hoffman 5028146e8c added __slots__ to decrease memory footprint of Link objects 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40413
 2008-11-25 00:39:11 +00:00
elpolilla a8903d5877 Changed deletion of _adaptor_dict attribute in parse command that could trigger exceptions 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40412
 2008-11-24 18:31:05 +00:00