diff --git a/sep/sep-011.rst b/sep/sep-011.rst new file mode 100644 index 000000000..3498d3950 --- /dev/null +++ b/sep/sep-011.rst @@ -0,0 +1,36 @@ +======= ================================ +SEP 11 +Title Process models for Scrapy +Created 2009-11-16 +Status Partially implemented - see #168 +======= ================================ + +================================== +SEP-011: Process models for Scrapy +================================== + +There is an interest of supporting different process models for Scrapy, mainly +to help prevent memory leaks which affect running all spiders in the same +process. + +By running each spider on a separate process (or pool of processes) we'll be +able to "recycle" process when they exceed a maximum amount of memory. + +Supported process models +======================== + +The user could choose between different process models: + +1. in process (only method supported so far) +2. pooled processes (a predefined pool of N processes, which could run more than one spider each) +3. separate processes (one process per spider) + +Using different processes would increase reliability at the cost of performance. + +Another ideas to consider +========================= + +- configuring pipeline process models - so that we can have a process exclusive + for running pipelines +- support writing spidersr in different languages when we don't use an in + process model