From 9a52adce571cff93f0eed1cf92f3b0a5093bddeb Mon Sep 17 00:00:00 2001 From: Edwin O Marshall Date: Thu, 6 Mar 2014 17:06:20 -0500 Subject: [PATCH 1/2] sep 11 for #629 --- sep/sep-011.rst | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 sep/sep-011.rst diff --git a/sep/sep-011.rst b/sep/sep-011.rst new file mode 100644 index 000000000..3498d3950 --- /dev/null +++ b/sep/sep-011.rst @@ -0,0 +1,36 @@ +======= ================================ +SEP 11 +Title Process models for Scrapy +Created 2009-11-16 +Status Partially implemented - see #168 +======= ================================ + +================================== +SEP-011: Process models for Scrapy +================================== + +There is an interest of supporting different process models for Scrapy, mainly +to help prevent memory leaks which affect running all spiders in the same +process. + +By running each spider on a separate process (or pool of processes) we'll be +able to "recycle" process when they exceed a maximum amount of memory. + +Supported process models +======================== + +The user could choose between different process models: + +1. in process (only method supported so far) +2. pooled processes (a predefined pool of N processes, which could run more than one spider each) +3. separate processes (one process per spider) + +Using different processes would increase reliability at the cost of performance. + +Another ideas to consider +========================= + +- configuring pipeline process models - so that we can have a process exclusive + for running pipelines +- support writing spidersr in different languages when we don't use an in + process model From 2e92400e91da207e18d2d743d53c415121210d04 Mon Sep 17 00:00:00 2001 From: Edwin O Marshall Date: Fri, 7 Mar 2014 09:12:21 -0500 Subject: [PATCH 2/2] removed trac file --- sep/sep-011.trac | 30 ------------------------------ 1 file changed, 30 deletions(-) delete mode 100644 sep/sep-011.trac diff --git a/sep/sep-011.trac b/sep/sep-011.trac deleted file mode 100644 index 0a475bd18..000000000 --- a/sep/sep-011.trac +++ /dev/null @@ -1,30 +0,0 @@ -= SEP-011: Process models for Scrapy = - -[[PageOutline(2-5,Contents)]] - -||'''SEP:'''||11|| -||'''Title:'''||Process models for Scrapy|| -||'''Author:'''||Pablo Hoffman|| -||'''Created:'''||2009-11-16|| -||'''Status'''||Partially implemented - see #168 || - -== Introduction == - -There is an interest of supporting different process models for Scrapy, mainly to help prevent memory leaks which affect running all spiders in the same process. - -By running each spider on a separate process (or pool of processes) we'll be able to "recycle" process when they exceed a maximum amount of memory. - -== Supported process models == - -The user could choose between different process models: - - 1. in process (only method supported so far) - 2. pooled processes (a predefined pool of N processes, which could run more than one spider each) - 3. separate processes (one process per spider) - -Using different processes would increase reliability at the cost of performance. - -== Another ideas to consider == - - * configuring pipeline process models - so that we can have a process exclusive for running pipelines - * support writing spidersr in different languages when we don't use an in process model