RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,759 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

1759 Commits

Author SHA1 Message Date
Pablo Hoffman f565127eb2 added some missing file to MANIFEST.in 2009-09-28 23:55:00 -03:00
Pablo Hoffman 4a9d9282bc removed obsolete scrapy architecture dia diagram 2009-09-28 23:54:01 -03:00
Pablo Hoffman 7e12d3d41b removed old adaptors tests sample data 2009-09-28 23:16:17 -03:00
Pablo Hoffman 7428e4cebf splitted make_release.sh in: build_release.sh and sign_release.sh 
--HG--
rename : extras/make_release.sh => extras/build_release.sh
 2009-09-28 23:07:00 -03:00
Pablo Hoffman 504293b8c4 renamed tag: 0.7.0-rc1 to 0.7-rc1 2009-09-28 22:29:23 -03:00
Pablo Hoffman 2be739964d bumped version to 0.7 final 2009-09-28 22:28:49 -03:00
Daniel Grana 1f9e0f10eb don't log ignored requests errors at images pipeline 2009-09-28 12:38:49 -03:00
Daniel Grana c795771282 remove unused line from project settings template 2009-09-23 14:53:40 -03:00
Daniel Grana dfd35c5280 Add common javascript mimetypes to be detected as TextResponses, also improve mimetype detection to avoid querying twice classes dict 2009-09-23 14:26:58 -03:00
Ismael Carnales ef0e5d368f updated spider templates to reflect the new spider callback return policy 2009-09-22 11:25:42 -03:00
Ismael Carnales 5862ba7db7 modified doc to reflect the new spider callback return policy (lists not needed) 2009-09-22 11:25:40 -03:00
Ismael Carnales 802f918b69 removed obsolete doc static file 2009-09-22 11:25:38 -03:00
Daniel Grana 6467c7055d check iterable output of crawl spider and add tests for common function used to iterate spider output 2009-09-21 13:57:16 -03:00
Pablo Hoffman b6155801f0 Added tag 0.7.0-rc1 for changeset f4a14daa04e6 2009-09-18 00:52:41 -03:00
Pablo Hoffman bc778a0e5b make_release.sh: added warning about broken bdist_wininst 2009-09-17 15:28:36 -03:00
Daniel Grana 1b28858bf9 improve XmlRpcRequest and update tests to cover more cases 2009-09-17 14:26:42 -03:00
Pablo Hoffman 48c3fb3ace made XmlRpcRequestTest inherit from RequestTest and removed redundant test 2009-09-17 12:21:43 -03:00
Pablo Hoffman 6845505e7a added make_release.sh script 2009-09-17 11:46:47 -03:00
Pablo Hoffman 6e10914454 added pypi classifiers to setup.py 2009-09-17 11:22:48 -03:00
Pablo Hoffman 6e93872955 updated installation guide for using releases 2009-09-17 11:06:55 -03:00
Pablo Hoffman 653db1d673 added bitmap for windows installer 2009-09-17 02:01:40 -03:00
Pablo Hoffman 132557dd14 some deployment changes in preparation for the 0.7.0 release candidate 2009-09-16 22:40:36 -03:00
Ismael Carnales fd41f06056 added doc on how to enable an Item Pipeline component 2009-09-16 14:19:16 -03:00
Ismael Carnales 404e7e09d7 changed spider doc references in BaseSpider class 2009-09-16 14:10:11 -03:00
Daniel Grana 062730cbd8 fix csv exporter documentation 2009-09-16 00:17:50 -03:00
Daniel Grana 08aedcbe76 Automated merge with ssh://hg.scrapy.org/scrapy 2009-09-15 09:35:24 -03:00
Daniel Grana e8514445dc bugfix image thumb when no thumb is configured 2009-09-15 09:35:21 -03:00
Pablo Hoffman 02e228ad76 added support for returning deferreds in spider manager close_domain() method, and making sure engine_stopped signal is always sent (even when no spiders have run) 2009-09-15 09:27:30 -03:00
Pablo Hoffman 66ff3b3013 renamed defer_failed to defer_fail 2009-09-15 08:44:31 -03:00
Pablo Hoffman 56b292e057 XmlItemExporter: added built-in support for exporting multi-valued fields (for convenience) 2009-09-14 22:05:52 -03:00
Pablo Hoffman e8960bf616 added runspider command to run spiders directly, without having to create a project 2009-09-14 22:05:14 -03:00
Pablo Hoffman fcbbb5001e ported spiderctl web console extensin to work with new core based on spider references 2009-09-14 20:35:47 -03:00
Pablo Hoffman 47aa716630 adapted web console to use unix timestamps for uptime instead of datetime 2009-09-14 20:35:07 -03:00
Pablo Hoffman bc463bc9e8 using time.time() instead of datetime.utcnow() in engine.start_time atttribute 2009-09-14 20:28:46 -03:00
Pablo Hoffman 7e07f76edd made pending_spiders attribute protected in spider scheduler 2009-09-14 20:27:54 -03:00
Pablo Hoffman 9b68432624 commented out line that was preventing errors from propagating to the request errback 2009-09-14 12:27:29 -03:00
Pablo Hoffman 2322312c63 Logging requests instead of responses in 'Crawled ...' messages 2009-09-14 10:31:29 -03:00
Pablo Hoffman 6f64dfe579 renamed spider manager close_domain() method to close_spider() 2009-09-14 10:06:54 -03:00
Pablo Hoffman 99467d4e6e Changed (unstable) scheduler middleware API to receive spider (instead of domain) in enqueue_request method 2009-09-13 20:51:43 -03:00
Pablo Hoffman 00873cd16c Another Spider Manager simplification: removed add_spider() method 2009-09-12 21:27:58 -03:00
Pablo Hoffman dc82550058 Do not impose an arbitrary encoding in spider templates, because we don't know beforehand what enconding our users will use in their editors. 2009-09-12 21:04:21 -03:00
Pablo Hoffman 58bdd7303c fixed deprecated import in 'crawl' spider template (thanks Anibal) 2009-09-12 20:59:04 -03:00
Pablo Hoffman 90407d2789 added missing colon 2009-09-12 19:33:16 -03:00
Pablo Hoffman 3f30fee6ea added first (not yet stable) revision of Crawler class, which allows to use the Scrapy crawler from stand-alone scripts 2009-09-12 19:32:23 -03:00
Pablo Hoffman 1381c1e50a removed (no longer needed) hack in setup.py 2009-09-12 14:50:05 -03:00
Pablo Hoffman 921fc4f3bf Big Scrapy core refactoring to pass around spider references instead of domains. 
This is to avoid accessing the scrapy.spider.spiders singleton for "resolving"
spiders, which is considered an "evil" practice because it ties us to the
singleton model for the spider resolver, which is a bad thing.

This change will also work as the foundation for the API cleaning that we'll
perform for 0.8. We decided to introduce this change now to have a more common
basecode between 0.7 and 0.8, which will allow us to better support 0.7 until
0.8 is released.

However, this change doesn't modify the stable/documented API, nor does it
change the core logic. Those changes will land on the 0.8 branch, after 0.7 is
released.

--HG--
rename : scrapy/contrib/domainsch.py => scrapy/contrib/spiderscheduler.py
 2009-09-12 14:34:18 -03:00
Pablo Hoffman 655cfe138d removed unused imports 2009-09-11 19:38:31 -03:00
Pablo Hoffman 0c292e3350 removed hacky --callback option to crawl command 2009-09-11 19:36:00 -03:00
Pablo Hoffman e854d0d6ef removed redundant --nopipelines function. same behaviour can be obtained by clearing the ITEM_PIPELINES setting 2009-09-11 19:32:05 -03:00
Pablo Hoffman 8d49dc2fb5 changed IMAGES_THUMBS setting to a dict instead of a list of tuples, and more improvements to images pipeline doc 2009-09-11 17:36:00 -03:00