RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

removed trac file

This commit is contained in:
Edwin O Marshall 2014-03-07 09:12:21 -05:00
parent 9a52adce57
commit 2e92400e91
1 changed files with 0 additions and 30 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
sep/sep-011.trac
View File

@ -1,30 +0,0 @@
= SEP-011: Process models for Scrapy =
[[PageOutline(2-5,Contents)]]
||'''SEP:'''||11||
||'''Title:'''||Process models for Scrapy||
||'''Author:'''||Pablo Hoffman||
||'''Created:'''||2009-11-16||
||'''Status'''||Partially implemented - see #168 ||
== Introduction ==
There is an interest of supporting different process models for Scrapy, mainly to help prevent memory leaks which affect running all spiders in the same process.
By running each spider on a separate process (or pool of processes) we'll be able to "recycle" process when they exceed a maximum amount of memory.
== Supported process models ==
The user could choose between different process models:
1. in process (only method supported so far)
2. pooled processes (a predefined pool of N processes, which could run more than one spider each)
3. separate processes (one process per spider)
Using different processes would increase reliability at the cost of performance.
== Another ideas to consider ==
* configuring pipeline process models - so that we can have a process exclusive for running pipelines
* support writing spidersr in different languages when we don't use an in process model