First, lets define what I mean by expansive bloat and state some facts about SBS 2003.

I define expansive bloat as an excessively big file that is growing quickly and out of control. In this case the Information Store (IS) was 200Gb and growing at about 5 Gb per hour.

SBS 2003 with the appropriate patches etc can only handle stores up to 75 Gb. After this, the mailbox stores are dismounted and email access is not available. This is a limitation hardwired into SBS 2003.

Imagine my surprise when a clients store dismounted (As expected) because the store was 200 Gb and climbing.

Looking in the MDBdata folder, I could see the transaction logs appearing and being culled (Circular logging was turned on) before my very eyes. The store size looked like a cars Odometer and ticking over.

Looking in the Exchange system manager (ESM) and adding up all the mailboxes sizes, comes to just under 6 Gb.

In an effort to reign in this bloat, I turned off mailbox and item retention. I checked the backups and trawled the exchange logs and eventlogs.

There are no strange event logs except to state that the store was too big and dismounted.

I looked at the SMTP incoming queue (in the mailroot folder) and the server outbound queues (in the ESM). They were empty. I looked at the MAPI connections to see if someone at a workstation was doing something odd, nothing.

My natural instinct was to freeze outgoing email and stop the Simple Mail Transport Protocol (SMTP). I restarted Exchange and dismounted and remounted the store. The only thing that stopped the Store expanding was turning off the SMTP service. This was my first clue. There was something stuck in the queues that I can’t see. This data is likely in the transaction logs.

I finally found a workstation with 2 stuck items in the Outbox that would not send. It kept retrying. I could not delete the email. Outlook reported that the MAPI was in an inconsistent state (And the recipients were getting the email over and over again). I logged in as this user using Outlook Web Access, that online version of the Outbox was empty. I then worked out that Outlook was using an OST and these two emails were  only in the OST version of the mailbox.

Simple fix. Delete the OST file.

Now I have a 200 Gb IS that will not remain mounted. I found the normal defrag informational alerts in the eventlog. The online Defrag was only clearing 1 Mb of whitespace, daily. Obviously something more serious is wrong in the store.

I attempted an offline defrag and repair with ESEutil (As Isinteg would not even talk to the store). It got to 50% and then failed with an error. The temporay file it created was 100Gb. I figured this means if it had gone to 100%, I would still have a 200 Gb store. The error provided no help. No One on Google or in the Technet forums had previously reported this error so I had nothing to work with.

Operation terminated with error -1014 (JET_errOutOfBuffers, Out of database page buffers

I spoke to the client who decided it was best to try to either try to recover mailboxes from the Exchange Recovery Group or, convert OST’s to PST’s and create a new store or (Using Recovery Toolbox for Outlook), recover from backups (Which I then found out were not as current as first expected) or use a tool like OnTrack PowerControls to dump out the email store mailboxes as PST files.

We attempted to use the Recovery Group however Exmerge could talk to the store but not see any mailboxes. This was a shame as the store mounted first time and looked fine.

In the end we created a new store and this did not bloat. We were able to reimport email from the OST files and the client is up and running.

One thing to remember, if you delete the store and create a new one, all premissions in the old store including delegates, needs to be recreated.

It is also worth nothing, Outlook users will get the error

Your mailbox has been temporarily moved on the MS Exchange server. A temporary mailbox exists, but might not have all of your previous data. You can connect to the temporary mailbox or work offline with all of your old data. If you choose to work with your old data you cannot send or receive email messages.

The fix is to delete the email profile from the mail applet in Control Panel and recreate it. Take note of any additional mailboxes and accounts before you do that. Also, if the user has PST files, make sure that they know their passwords (else use a tool like the PST password tool from Nirsoft). 

Go to Control Panel-> Mail-> Profiles-> Show profiles.
Highlight the current Exchange profile you are using and click Remove. 

Click YES when you are asked if you are sure.

To close the window, click OK.

Open Windows Explorer.

Browse to the folder C:\Documents and Settings\user\Local Settings\Application Data\Microsoft\Outlook.  Highlight and delete the Outlook.ost file (I prefer to rename it).

Create a new profile and if you use cached exchange mode, it will create a new .ost file. 

Please be aware that The Outlook.ost file is in a folder that is “hidden” by default.  If you do not see the folder, you may need to change the view settings in Explorer, as follows: 

Go to Tools>Folder options, View tab
Check the radio button for Show hidden files, folders, and drives
Uncheck the box for Hide protected operating system files (recommended) 
Click YES when you are asked if you are sure.
To close the window, click Apply and OK.

 So we have arrived at a better place with a much smaller information store.


Tags: , , , , ,