RPC averaged latency performance counter really high…

Hi folks,

Today I got a weird issue that took me quite a long time to figure it out and I would like to share with you guys. Everything started when a brand new DAG was deployed and everything went just fine during the installation process however during the initial testing I noticed a RPC averaged latency performance counter really high.

In the figure below you can see how bad was it and I the server had just a few users that were part of my pre-pilot phase.

I’ve done a couple of tests, as follows:

– First, I ran the Get-MailboxDatabaseCopyStatus and everything was golden

– I moved the databases to a single node, and then noticed that the utilization kept the same

– I restarted the RPC Client access and the same issue were the performance counter didn’t change (expected)

– I restarted the Information Store and then the performance counter went to 0 but as soon as I moved any DB the number would increase again.

– I ran Test-ReplicationHealth and no issues at all

– I use PAL tool to analyse performance and the disks were just fine

– I ran netstat to see the connections and nothing was outstanding

– Exchange Best Practices Analyzer

In this environment that I was working they have a LTM (Local Traffic Manager) and then I noticed that I could access the Domain Controller (\\unc) however I couldn’t ping. I double check and there was a SNAT and a Virtual Server were in place, the missing piece was the ability to ping the Domain Controllers, so in a F5 we should go to System / Configuration / Local Traffic and change the default behaviour of the box to All traffic in the SNAT Packet Forwarding setting.

After performing that change I moved the databases around and the RPC averaged latency came back to 0

Cheers,
Anderson Patricio
http://www.andersonpatricio.ca
http://www.andersonpatricio.org (Portuguese)
Twitter: @apatricio