RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,726 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

1726 Commits

Author SHA1 Message Date
Pablo Hoffman bc463bc9e8 using time.time() instead of datetime.utcnow() in engine.start_time atttribute 2009-09-14 20:28:46 -03:00
Pablo Hoffman 7e07f76edd made pending_spiders attribute protected in spider scheduler 2009-09-14 20:27:54 -03:00
Pablo Hoffman 9b68432624 commented out line that was preventing errors from propagating to the request errback 2009-09-14 12:27:29 -03:00
Pablo Hoffman 2322312c63 Logging requests instead of responses in 'Crawled ...' messages 2009-09-14 10:31:29 -03:00
Pablo Hoffman 6f64dfe579 renamed spider manager close_domain() method to close_spider() 2009-09-14 10:06:54 -03:00
Pablo Hoffman 99467d4e6e Changed (unstable) scheduler middleware API to receive spider (instead of domain) in enqueue_request method 2009-09-13 20:51:43 -03:00
Pablo Hoffman 00873cd16c Another Spider Manager simplification: removed add_spider() method 2009-09-12 21:27:58 -03:00
Pablo Hoffman dc82550058 Do not impose an arbitrary encoding in spider templates, because we don't know beforehand what enconding our users will use in their editors. 2009-09-12 21:04:21 -03:00
Pablo Hoffman 58bdd7303c fixed deprecated import in 'crawl' spider template (thanks Anibal) 2009-09-12 20:59:04 -03:00
Pablo Hoffman 90407d2789 added missing colon 2009-09-12 19:33:16 -03:00
Pablo Hoffman 3f30fee6ea added first (not yet stable) revision of Crawler class, which allows to use the Scrapy crawler from stand-alone scripts 2009-09-12 19:32:23 -03:00
Pablo Hoffman 1381c1e50a removed (no longer needed) hack in setup.py 2009-09-12 14:50:05 -03:00
Pablo Hoffman 921fc4f3bf Big Scrapy core refactoring to pass around spider references instead of domains. 
This is to avoid accessing the scrapy.spider.spiders singleton for "resolving"
spiders, which is considered an "evil" practice because it ties us to the
singleton model for the spider resolver, which is a bad thing.

This change will also work as the foundation for the API cleaning that we'll
perform for 0.8. We decided to introduce this change now to have a more common
basecode between 0.7 and 0.8, which will allow us to better support 0.7 until
0.8 is released.

However, this change doesn't modify the stable/documented API, nor does it
change the core logic. Those changes will land on the 0.8 branch, after 0.7 is
released.

--HG--
rename : scrapy/contrib/domainsch.py => scrapy/contrib/spiderscheduler.py
 2009-09-12 14:34:18 -03:00
Pablo Hoffman 655cfe138d removed unused imports 2009-09-11 19:38:31 -03:00
Pablo Hoffman 0c292e3350 removed hacky --callback option to crawl command 2009-09-11 19:36:00 -03:00
Pablo Hoffman e854d0d6ef removed redundant --nopipelines function. same behaviour can be obtained by clearing the ITEM_PIPELINES setting 2009-09-11 19:32:05 -03:00
Pablo Hoffman 8d49dc2fb5 changed IMAGES_THUMBS setting to a dict instead of a list of tuples, and more improvements to images pipeline doc 2009-09-11 17:36:00 -03:00
Pablo Hoffman e20f766792 fixed some typos 2009-09-11 16:55:37 -03:00
Pablo Hoffman c2fe350f72 more changes to images pipeline doc 2009-09-11 16:53:36 -03:00
Ismael Carnales ada46a2dbb styled imagesp doc 2009-09-11 15:30:46 -03:00
Pablo Hoffman be0f2beef0 more cleanup to scheduler middelware doc, and permanentely moved to experimental doc 2009-09-11 13:27:31 -03:00
Pablo Hoffman 0af052b68f removed confusing title 2009-09-11 12:19:18 -03:00
Pablo Hoffman f3240748cb changed link to scheduler middleware doc, now in experimental 2009-09-11 12:03:23 -03:00
Ismael Carnales 3998a0cb58 added more scheduler middleware documentation, and moved it to experimental 
--HG--
rename : docs/topics/scheduler-middleware.rst => docs/experimental/scheduler-middleware.rst
 2009-09-11 11:58:53 -03:00
Pablo Hoffman d242a20573 updated images pipeline doc 2009-09-11 11:47:12 -03:00
Daniel Grana 40d38b18d8 Automated merge with ssh://hg.scrapy.org/scrapy 2009-09-11 11:04:47 -03:00
Daniel Grana 96bc6780d3 imagespipeline: change scraped_url to url 2009-09-11 11:03:53 -03:00
Pablo Hoffman 0174bee4bc simplified implementation of scrapy.fetcher 2009-09-10 19:27:47 -03:00
Pablo Hoffman f1bb8dc2a3 first cleanup of spider manager api 
- removed asdict() and reload() methods
- added list() method
- removed default spider
 2009-09-10 19:06:46 -03:00
Pablo Hoffman f85813cd94 added FAQ entry about scrapy recipes and community spiders 2009-09-10 18:32:50 -03:00
Daniel Grana 734464825b allow to override httpclientfactory 2009-09-10 16:28:08 -03:00
Pablo Hoffman 269724a2b7 added Debugger extension, removed StackTraceDump from extensions available by default 2009-09-08 22:32:17 -03:00
Pablo Hoffman 2974c2c4b5 some additional checks on using unicode url/body in Request/Response objects 2009-09-07 15:20:41 -03:00
Pablo Hoffman 7a88c0d8e5 shell: fixed bug when typing exit() in python console - fixes #103 2009-09-07 14:57:50 -03:00
Ismael Carnales 4ddfa9a2a3 stlyed downloaded middleware doc 2009-09-07 12:18:57 -03:00
Ismael Carnales e3df11e5bb added module directive to spidermw documentation 2009-09-07 12:03:24 -03:00
Ismael Carnales 30c2ad3f0c added urllength spider middleware test 2009-09-07 11:14:47 -03:00
Ismael Carnales c4ad2bea5d added urlfilter spidermw test 2009-09-07 11:14:47 -03:00
Ismael Carnales 43bd00dea2 added referer spider middleware test 2009-09-07 11:14:46 -03:00
Ismael Carnales 1c700749ae added offside spider middleware test 2009-09-07 11:14:45 -03:00
Ismael Carnales 083635ebaf added depth spider middleware test 2009-09-07 11:14:43 -03:00
Ismael Carnales f0b5892aa5 fixed stats downloadermw test 2009-09-07 11:14:38 -03:00
Pablo Hoffman 82ca5e26f5 renamed test_xpath.py to test_selector.py 
--HG--
rename : scrapy/tests/test_xpath.py => scrapy/tests/test_selector.py
 2009-09-07 09:38:24 -03:00
Pablo Hoffman 4023914b10 added support for instantiating TextResponse (or any subclass) with unicode urls, improved organization of request/response unittests 2009-09-07 09:37:46 -03:00
Pablo Hoffman e2bd1be995 better aws code arrangement 
--HG--
rename : scrapy/tests/test_aws.py => scrapy/tests/test_utils_aws.py
 2009-09-04 18:07:51 -03:00
Pablo Hoffman 827aa19c6e removed obsolete scrapy.utils.db module 2009-09-04 17:38:14 -03:00
Pablo Hoffman 7466150286 removed some more obsolete middlewares 2009-09-04 17:32:34 -03:00
Pablo Hoffman 861a803cc3 removed obsolete RestrictMiddleware 2009-09-04 17:22:56 -03:00
Pablo Hoffman 0631102153 removed backwards compatibility for old errorpages downloader middlware 2009-09-04 17:19:03 -03:00
Ismael Carnales 043e7355f7 added some missing spidermw tests 2009-09-04 14:11:56 -03:00