diff --git a/sep/sep-011.rst b/sep/sep-011.rst new file mode 100644 index 000000000..3498d3950 --- /dev/null +++ b/sep/sep-011.rst @@ -0,0 +1,36 @@ +======= ================================ +SEP 11 +Title Process models for Scrapy +Created 2009-11-16 +Status Partially implemented - see #168 +======= ================================ + +================================== +SEP-011: Process models for Scrapy +================================== + +There is an interest of supporting different process models for Scrapy, mainly +to help prevent memory leaks which affect running all spiders in the same +process. + +By running each spider on a separate process (or pool of processes) we'll be +able to "recycle" process when they exceed a maximum amount of memory. + +Supported process models +======================== + +The user could choose between different process models: + +1. in process (only method supported so far) +2. pooled processes (a predefined pool of N processes, which could run more than one spider each) +3. separate processes (one process per spider) + +Using different processes would increase reliability at the cost of performance. + +Another ideas to consider +========================= + +- configuring pipeline process models - so that we can have a process exclusive + for running pipelines +- support writing spidersr in different languages when we don't use an in + process model diff --git a/sep/sep-011.trac b/sep/sep-011.trac deleted file mode 100644 index 0a475bd18..000000000 --- a/sep/sep-011.trac +++ /dev/null @@ -1,30 +0,0 @@ -= SEP-011: Process models for Scrapy = - -[[PageOutline(2-5,Contents)]] - -||'''SEP:'''||11|| -||'''Title:'''||Process models for Scrapy|| -||'''Author:'''||Pablo Hoffman|| -||'''Created:'''||2009-11-16|| -||'''Status'''||Partially implemented - see #168 || - -== Introduction == - -There is an interest of supporting different process models for Scrapy, mainly to help prevent memory leaks which affect running all spiders in the same process. - -By running each spider on a separate process (or pool of processes) we'll be able to "recycle" process when they exceed a maximum amount of memory. - -== Supported process models == - -The user could choose between different process models: - - 1. in process (only method supported so far) - 2. pooled processes (a predefined pool of N processes, which could run more than one spider each) - 3. separate processes (one process per spider) - -Using different processes would increase reliability at the cost of performance. - -== Another ideas to consider == - - * configuring pipeline process models - so that we can have a process exclusive for running pipelines - * support writing spidersr in different languages when we don't use an in process model