Publishing many items uses a lot of memory - tridion

Question

I have a problem similar to this, although hopefully I've narrowed it down a little.
I am currently testing the publishing part of my Tridion 2011 installation (Conent Manager and Content Delivery on separate servers). When I try to publish many pages at once (1000 in my case) the process dllhst3g.exe *32 starts to acquire a large amount of memory. This is understandable as there is a lot of work to do but it never gives it back. This is causing the content manager to run slowly and eventually crash.
I was getting out of memory exceptions being thrown by the Tridion Content Manager when the dllhst3g.exe *32 process had around 3.6GB of memory (I assume this is due to it being a 32 bit process). To prevent these I limited the SDL Tridion Content Manager COM+ Application to an arbitrary 500Mb per process which gets forcibly terminated 15 minutes after the limit is reached. This has stopped the out of memory exceptions for now but I still have performance issues and a potential to run out of memory if more than my servers 8GB is allocated in the 15 minute window. More about this here
I have ruled out the underlying PageTemplate code as the associated page template has no code associated. These pages are blank.
The issue also seems to be much more prominent when using the Core Service API. Code I am using is
using (var client = new Tridion2011CoreService.CoreServiceClient())
{
foreach (var id in ids) // ids is a collection of 500 page ids
{
// publishing to staging and live
var targets = new string[] { "tcm:0-7-65538", "tcm:0-8-65538" };
var publishInstructionData = new PublishInstructionData();
publishInstructionData.ResolveInstruction = new ResolveInstructionData();
publishInstructionData.RenderInstruction = new RenderInstructionData();
var readOptions = new ReadOptions();
client.Publish(new string[] { id }, publishInstructionData, targets, PublishPriority.Normal, readOptions);
}
}
(I realise I could send all the ids through with one call to publish but then I hit a message limit error and as far as I am aware the result of multiple calls and single calls is effectively the same)
Any ideas?
(let me know if I have missed out any details and I'll update the question accordingly)
Forgot to add. The Content Manager is installed on a virtualised Windows Server 2008 with 4 CPUs, 8GB RAM and 50GB disk space. The Content Delivery is installed on a separate server with the same specs.

Answer 1

After contacting SDL customer support I was told that the only solution is to recycle the dllhost process when it takes up a pre-determined amount of memory.
To do this for SDL Tridion Content Manager 2011 running on Windows Server 2008
Open Component Services (either search for it or Control Panel -> Administrative Tools)
Expand the tree Component Services -> Computers -> My Computer -> COM+ Applications
Right click SDL Tridion Content Manager and select Properties
Select the Pooling & Recycling tab
Set Pool Size to 1
Set Lifetime limit to 0
Set Memory Limit to 524288 (512 MB) or choose your own limit (I'd go for a value less than 1GB)
Set Call Limit to 0
Set Activation Limit to 0
If you can, restart your system. Otherwise, restart all the Tridion Services (Control Panel -> Administrative Tools -> Services and restart everything that starts with "Tridion")
Maybe the "real" fix is but a patch away...

Answer 2

Try monitoring the transport package root folder. This would be set in SDL Tridion MMC snap in. Default value would be C:\temp in CMS server. Please check the work area folder of transport which would be set in cd_transport config xml in Tridion installed folder\Config. Also check for the deployer incoming and work folders.Large volume of files accumlating in these folders may cause in poor performance and try cleaning up these folders if you have errored out folders or files to prevent Tridion from retrying the failed transactions. Also if your deployer is runnning as a website-httpupload.asp, try verifying the application domain recycle settings of deployer just to make sure you're not resetting appdomain every few minutes.
You could also try looking in to related items of your publish items.If you have lot of versions or unwanted related items, you could think of custom resolving of publish items to avoid unwanted components with a particular schema etc.Nuno has provided a very good
article in below link. Thanks to Nuno.. http://nunolinhares.blogspot.nl/2011/10/tridion-publisher-and-custom-resolvers.html