RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
79 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

79 Commits

Author SHA1 Message Date
Pablo Hoffman b45d87d0fe removed duplicated code 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4079
 2008-07-18 01:51:49 +00:00
olveyra cdb3b02510 fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4078
 2008-07-17 19:20:10 +00:00
olveyra 7c1ebfd16a option help fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4077
 2008-07-17 19:06:03 +00:00
olveyra d765b133f0 - --default-spider option gives the default spider (old --spider option) 
- --spider option now forces to use the given spider domain when
arguments are urls

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4076
 2008-07-17 19:00:05 +00:00
olveyra abde0e5f51 readed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4075
 2008-07-17 18:15:46 +00:00
olveyra fb43b935c6 deleted 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4074
 2008-07-17 18:14:52 +00:00
olveyra 6f028cc38a moved cluster-ctl script to scrapy branch 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4073
 2008-07-17 18:09:32 +00:00
olveyra f515942f54 - added verbosity levels 
- now log paths includes a folder named by the date, so it is easier
to mantain logs in server

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4072
 2008-07-17 15:41:15 +00:00
olveyra 5403fd3d9f - added disable_node and enable_node functions 
- removed unused imports
- autoreload of lost nodes
- some code improvements and fixes

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4071
 2008-07-16 19:17:46 +00:00
Ezequiel Rivero 8150fd6a16 size fix for menu in trac 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4070
 2008-07-16 17:50:58 +00:00
olveyra e8eab24dc0 don't update status in run callback, so to avoid lots of bouncing 
domains. The domains will be loaded softly on each node update

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4069
 2008-07-16 12:09:42 +00:00
olveyra 338a485bc1 - load only one domain per node (and load the following when the run 
callback is executed). This way, we avoid to load lot of domains that
will bounce. Also, we mix up better the domains between available nodes.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4068
 2008-07-16 11:20:53 +00:00
olveyra ace6e3c430 - Reschedule a domain that is already running or loading in some node 
- No schedule a domain that is already scheduled

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4067
 2008-07-15 18:53:37 +00:00
olveyra 69d1c57692 small fix and pepocho fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4066
 2008-07-15 18:06:14 +00:00
olveyra e71aa06f97 Check the node will not run a domain that is already running 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4065
 2008-07-15 12:11:32 +00:00
olveyra 2b6189a3f8 renamed setting SCRAPY_PICKLED_SETTINGS to 
SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4064
 2008-07-14 14:10:26 +00:00
olveyra 11bc122a54 pickled settings must go in overrides settings not default, because of 
precedence

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4063
 2008-07-14 13:39:58 +00:00
Pablo Hoffman 49175d1990 added new test case for xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4062
 2008-07-14 13:31:00 +00:00
Pablo Hoffman 28b5bb3240 added utf-16 (and other encodings) support to xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4061
 2008-07-14 13:22:05 +00:00
Pablo Hoffman 90e93a7635 imports should be at module's top 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4060
 2008-07-14 13:18:58 +00:00
olveyra 9b3c3c14e5 logs extensions NotConfigured exception message 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4059
 2008-07-14 12:42:58 +00:00
olveyra 377e17b7db More data in log message 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4058
 2008-07-14 12:35:46 +00:00
olveyra a4cfea54e8 fixed env name 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4057
 2008-07-11 19:34:39 +00:00
olveyra 1eb583a357 added log msg 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4056
 2008-07-11 19:31:08 +00:00
olveyra ba0f87ed2c changed CLUSTER_WORKER_LOGDIR to CLUSTER_LOGDIR 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4055
 2008-07-11 18:27:19 +00:00
olveyra 585f35fb3a Cluster Master improvements: 
- rescheduling now goes with original priority decreased by one
- Added GLOBAL_CLUSTER_SETTINGS
- Added PB remote method load_node so the worker also can initiate a
connection

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4054
 2008-07-11 16:21:23 +00:00
olveyra 0c35b7ed2a Now worker pass settings to process via SCRAPY_PICKLED_SETTINGS 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4053
 2008-07-11 16:08:28 +00:00
olveyra 8ea2dc94d7 renamed pickled data 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4052
 2008-07-11 16:05:20 +00:00
olveyra 94993b769a added SCRAPY_PICKLED_DEFAULT_SETTINGS, a string to pass by environment 
setting an arbitrary set of settings via "pickle" python module.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4051
 2008-07-11 15:59:08 +00:00
Daniel Grana c97e7d0daf show if crawled pages are cached or live versions 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4050
 2008-07-08 19:48:12 +00:00
olveyra 805eeec844 deleted unneeded log 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4049
 2008-07-08 18:08:20 +00:00
olveyra fc4e0eaa2a fix a bug with rescheduling a domain when no free slot are in worker 
(a domain was passed instead of a list, and that raises the one-letter
domains bug), also reschedule with priority=0)

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4048
 2008-07-08 18:03:20 +00:00
olveyra 863d94adec added CLUSTER_MASTER_CACHEFILE setting, changed SVN_WORKDIR to 
CLUSTER_WORKER_SVNWORKDIR

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4047
 2008-07-07 18:00:35 +00:00
olveyra e78f7a93bd better way to stop engine with process signals 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4046
 2008-07-07 17:51:09 +00:00
Pablo Hoffman 648b312bca engine.stop() is now always executed when reactor stops 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4045
 2008-07-07 17:37:55 +00:00
olveyra 6bdbcd8173 pep8 pal pablo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4044
 2008-07-04 19:02:19 +00:00
olveyra 1d876db80b pass to crawl process the correct python path 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4043
 2008-07-04 18:59:36 +00:00
olveyra 6bbbb0c1f6 master logs response from worker. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4042
 2008-07-04 13:23:21 +00:00
olveyra e11d824ed0 added --pidfile option 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4041
 2008-07-04 13:22:27 +00:00
olveyra 23b3408403 - added svn update support 
- removed passing of env variables
- added automatic group settings load

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4040
 2008-07-02 19:42:36 +00:00
olveyra 8344819a92 added persistence of pending list in master 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4039
 2008-07-02 15:07:29 +00:00
olveyra 9c713529e4 fix and webservice api doc update 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4038
 2008-07-01 15:28:07 +00:00
olveyra 74ef3ab5b0 added environment options 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4037
 2008-07-01 15:17:00 +00:00
olveyra d2684dc17b Ok, first functional version of pbcluster 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4036
 2008-07-01 13:17:23 +00:00
Pablo Hoffman d993f493b5 some improvments to XPathSelectors: 
- x() method now returns the same XPathSelector type of its parent
- added tests to check this
- added tests to verify that XML and HTML XPathSelector behave differently when
  parsing some non trivial markup

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4035
 2008-07-01 02:21:12 +00:00
Pablo Hoffman 1f7f0d0994 replaced XMLNodeIterator with xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4034
 2008-06-30 17:20:56 +00:00
Ezequiel Rivero 3ee14952aa css styling changes. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4033
 2008-06-30 15:07:27 +00:00
olveyra 8a81fe82c7 deleted prints 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4032
 2008-06-30 14:24:49 +00:00
olveyra 53838d6a7d pbcluster commit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4031
 2008-06-30 14:20:27 +00:00
Pablo Hoffman 4caadf6b67 added (yet another) xml node iterator based entirely in regex 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4030
 2008-06-29 06:08:48 +00:00