BlackBerry Enterprise Server and Exchange Server 2010 Throttling Policies

One of the new features in Exchange Server 2010 is the concept of Client Throttling Policies. In summary, Client Throttling Policies are designed to limit the amount of system resources a given user can consume and in turn impact performance for other Exchange users. Out of the box there is a default throttling policy (use the Get-ThrottlingPolicy cmdlet) applied to all users:

RunspaceId                     : ba3cdf92-fc9f-4a70-a912-2cf225e6d573
IsDefault                      : True
EASMaxConcurrency              : 10
EASPercentTimeInAD             :
EASPercentTimeInCAS            :
EASPercentTimeInMailboxRPC     :
EWSMaxConcurrency              : 10
EWSPercentTimeInAD             :
EWSPercentTimeInCAS            :
EWSPercentTimeInMailboxRPC     :
EWSMaxSubscriptions            :
EWSFastSearchTimeoutInSeconds  : 60
EWSFindCountLimit              :
IMAPMaxConcurrency             :
IMAPPercentTimeInAD            :
IMAPPercentTimeInCAS           :
IMAPPercentTimeInMailboxRPC    :
OWAMaxConcurrency              : 5
OWAPercentTimeInAD             :
OWAPercentTimeInCAS            :
OWAPercentTimeInMailboxRPC     :
POPMaxConcurrency              : 20
POPPercentTimeInAD             :
POPPercentTimeInCAS            :
POPPercentTimeInMailboxRPC     :
PowerShellMaxConcurrency       : 18
PowerShellMaxCmdlets           :
PowerShellMaxCmdletsTimePeriod :
ExchangeMaxCmdlets             :
PowerShellMaxCmdletQueueDepth  :
RCAMaxConcurrency              : 20
RCAPercentTimeInAD             :
RCAPercentTimeInCAS            :
RCAPercentTimeInMailboxRPC     :
MessageRateLimit               :
RecipientRateLimit             :
ForwardeeLimit                 :
CPUStartPercent                : 75
AdminDisplayName               :
ExchangeVersion                : 0.10 (14.0.100.0)
Name                           : DefaultThrottlingPolicy_f017f530-3edf-4c59-9955-d94bb7892fb0
DistinguishedName              : CN=DefaultThrottlingPolicy_f017f530-3edf-4c59-9955-d94bb7892fb0,CN=Global Settings,CN=
                                 GreenOrg,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=green,DC=briandesmond,D
                                 C=net
Identity                       : DefaultThrottlingPolicy_f017f530-3edf-4c59-9955-d94bb7892fb0
Guid                           : af1aeaac-4d88-43da-92df-24c0924d4ad8
ObjectCategory                 : green.briandesmond.net/Configuration/Schema/ms-Exch-Throttling-Policy
ObjectClass                    : {top, msExchGenericPolicy, msExchThrottlingPolicy}
WhenChanged                    : 10/10/2009 5:44:29 PM
WhenCreated                    : 10/10/2009 5:44:11 PM
WhenChangedUTC                 : 10/10/2009 10:44:29 PM
WhenCreatedUTC                 : 10/10/2009 10:44:11 PM
OrganizationId                 :
OriginatingServer              : BRIAN-GRDC02.green.briandesmond.net
IsValid                        : True

 

As you can see, most of the valuesa re null, however each of the services has a *MaxCurrency property. These define the maximum number of connections a given user can have to that service. For example, the EASMaxConcurrency value limits a given user to a maximum of ten Exchange ActiveSync connections. Each of the values is documented in the Set-ThrottlingPolicy cmdlet documentation. Of particular note to this discussion is the RCAMaxConcurrency value which defines the maximum number of concurrent connections a given user can have to the RPC Client Access service. The RPC Client Access service is new in Exchange 2010 and it handles all MAPI connections to Exchange.

BlackBerry Enterprise Server (BES) uses a single service account to proxy all of the connections to Exchange on behalf of BlackBerry users. The side effect of this is that it’s quite likely that BES will need to have more than twenty (default limit) connections open to Exchange at a given time. If you review the documentation from RIM, they recommend setting the RCAMaxConcurrency value to null (equivalent to unlimited) for all users. This is really not a great idea at all.

Instead, what you can do is define a new Client Throttling Policy without an RCAMaxConcurrency value and apply it directly to the BES service account. The PowerShell script below does just that. The script assumes that your BES service account is called “BESAdmin”. If it isn’t modify the script accordingly.

New-ThrottlingPolicy "BES Throttling Policy" -RCAMaxConcurrency:$null
Set-Mailbox besadmin -ThrottlingPolicy "BES Throttling Policy"


You can easily confirm that the new policy is applied to the BESAdmin account by inspecting the properties of BESAdmin mailbox:



Get-Mailbox besadmin | fl Name,ThrottlingPolicy


You should see results similar to the following:



Name             : BES Admin
ThrottlingPolicy : BES Throttling Policy

How to Create an ActiveSync Device Report

Exchange logs quite a bit of info about ActiveSync device partnerships and you can use this to create reports about the utilization of mobility features in your organization. Getting this data requires a couple of intermediate steps before you can export it to a CSV for processing in something like Excel (or another script). The PowerShell script below will export all of the ActiveSync device relationships in your organization. Keep in mind that this will include old relationships which are no longer active. Depending on how large your organization is and the number of device relationships out there, it may take a little while for the script to run.

Note: If you have a mixed version organization (e.g. Exchange 2007 and Exchange 2010), you’ll need to run the script twice. Once in the Exchange 2007 Management Shell and once in the Exchange 2010 Management Shell. The cmdlets used here are not backwards (or forward compatible). I’ve provided two versions of the script – one for Exchange 2007 and one for Exchange 2010.

Exchange 2007 Version

$devices = @()
$mailboxes = Get-CASMailbox -ResultSize:Unlimited | Where-Object {$_.HasActiveSyncDevicePartnership -eq $true -and $_.ExchangeVersion.ExchangeBuild -ilike "8*"}

foreach ($m in $mailboxes) 
{
	$devices += Get-ActiveSyncDeviceStatistics -Mailbox $m.Identity
}

$devices | Export-Csv DeviceStats.csv




Exchange 2010 Version



$devices = @()
$mailboxes = Get-CASMailbox -ResultSize:Unlimited | Where-Object {$_.HasActiveSyncDevicePartnership -eq $true -and $_.ExchangeVersion.ExchangeBuild -ilike "14*"}

foreach ($m in $mailboxes) 
{
	$devices += Get-ActiveSyncDeviceStatistics -Mailbox $m.Identity
}

$devices | Export-Csv DeviceStats.csv




You can open the exported CSV in Excel from here and generate reports based on that. There is quite a bit of information in the report including some personally identifiable information (PII) for the devices so keep that in mind before redistributing the raw data file.

Small Update to Redirection Blog

Last week, I posted about how to redirect HTTP connects to Exchange 2010 OWA to HTTPS. There was a small issue in the post which I’ve now corrected. If you explicitly disabled HTTP Redirection for the OWA virtual directory, you would break the /exchange, /public, and /exchweb virtual directories which redirect to /owa.

If you browse to https://owa.customer.com/exchange, you might see the following event in the Application log of your CAS server:

Log Name:      Application
Source:        ASP.NET 2.0.50727.0
Date:          1/31/2010 2:20:16 PM
Event ID:      1310
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      CAS01.green.briandesmond.net

Description:
Event code: 3008
Event message: A configuration error has occurred.
Event time: 1/31/2010 2:20:16 PM
Event time (UTC): 1/31/2010 10:20:16 PM
Event ID: 1dd0ff95241040a48b5acc09bff2e3ad
Event sequence: 32
Event occurrence: 31
Event detail code: 0 
Application information:    
Application domain: /LM/W3SVC/1/ROOT-2-129092586348966635    
Trust level: Full    
Application Virtual Path: /    
Application Path: C:\inetpub\wwwroot\    
Machine name: CAS01
Process information:    
Process ID: 2268    
Process name: w3wp.exe    
Account name: IIS APPPOOL\DefaultAppPool 
Exception information:    
Exception type: ConfigurationErrorsException    
Exception message: It is an error to use a section registered as allowDefinition=’MachineToApplication’ beyond application level.  This error can be caused by a virtual directory not being configured as an application in IIS. (C:\Exchange\ClientAccess\owa\web.config line 31) 
Request information:    
Request URL: http://localhost/exchange/default.aspx    
Request path: /exchange/default.aspx    
User host address: 127.0.0.1    
User:     
Is authenticated: False    
Authentication Type:     
Thread account name: IIS APPPOOL\DefaultAppPool 
Thread information:    
Thread ID: 19    
Thread account name: IIS APPPOOL\DefaultAppPool    
Is impersonating: False    
Stack trace:   
   at System.Configuration.ConfigurationSchemaErrors.ThrowIfErrors(Boolean ignoreLocal)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSection(String configKey)
   at System.Web.Configuration.RuntimeConfig.GetSectionObject(String sectionName)
   at System.Web.Configuration.RuntimeConfig.GetSection(String sectionName, Type type, ResultsIndex index)
   at System.Web.Configuration.RuntimeConfig.get_Identity()
   at System.Web.HttpContext.SetImpersonationEnabled()
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)

To resolve this, open the web.config file for OWA. The path is in the event. I highlighted it in red boldface font above so you know where to look. Inside of the web.config file, search for and remove this line:

<httpRedirect enabled="false" />

Warning: Improperly editing the web.config file for OWA could render it entirely inoperable. I highly recommend that you save a backup prior to making any changes to the file.

Redirecting OWA URLs in Exchange 2010

One of the things I’ve been doing for as long as I can remember is redirecting requests that don’t go to https://owa.customer.com/owa (or /exchange) to the correct URL. So, if someone goes to http://owa.customer.com or https://owa.customer.com, they get redirected to the correct (secure) URL. Historically I’ve always done this with two components:

  • A custom website listening on Port 80 on each CAS server
  • A default.aspx file in the root of the Default Web Site redirecting to /owa

This approach no longer works with Exchange 2010 CAS because the PowerShell virtual directory actually operates over Port 80 (authentication is Kerberized). If you try and tinker with this, you’ll start getting errors from Remote PowerShell like this:

VERBOSE: Connecting to cas01.customer.com
[cas01.customer.com] The WinRM service cannot process the request because the request needs to be sent to a different machine. Use the redirect information to send the request to a new machine.  Redirect location reported: https://owa.customer.com/owa/PowerShell. To automatically connect to the redirected URI, verify "MaximumConnectionRedirectionCount" property of session preference variable "PSSessionOption" and use "AllowRedirection" parameter on the cmdlet.
    + CategoryInfo          : OpenError: (System.Manageme….RemoteRunspace:RemoteRunspace) [], PSRemotingTransportRedirectException
    + FullyQualifiedErrorId : PSSessionOpenFailed

In order to work around this, you need to use the HTTP Redirection feature in IIS (the default.aspx trick mentioned above should work too), as well as remove the requirement for SSL at the top level Default Web Site object. You have to be careful doing this because when you set settings on the web site, IIS will push them down to any virtual directory below which does not explicitly set that setting itself. To setup the redirect, select the Default Web Site in IIS Manager, and open the HTTP Redirect option under IIS. Complete it like this:

image

Warning: It’s very important that you check the second checkbox as shown in the screenshot above!

Once this step is complete, you need to remove the enforced redirect from each of the virtual directories under the Default Web Site. To do this, select each virtual directory individually, and then open the HTTP Redirect property and uncheck the “Redirect requests to this destination” checkbox. You’ll need to do this on the following virtual directories:

  • aspnet_client
  • Autodiscover
  • ecp
  • EWS
  • Microsoft-Server-ActiveSync
  • OAB
  • owa
  • PowerShell
  • Rpc

Note: The Exchange, Exchweb, and Public virtual directories should redirect to /owa.

If at this point you simply browse to http://cas01.customer.com, you’ll get an HTTP 403.4 error. This is because SSL is required at the top-level website. In order to get the redirect working, we need to disable SSL for the toplevel website while leaving it enabled for the relevant child virtual directories.

Select the Default Web Site and open the SSL Settings properties. Uncheck the Require SSL checkbox as shown below:

image

Like the redirection settings, this change will be inherited down the tree for any virtual directory which does not explicitly set the setting independently. Ensure that SSL is required for the following virtual directories:

  • Autodiscover
  • ecp
  • EWS
  • Microsoft-Server-ActiveSync
  • OAB
  • owa
  • Rpc

Warning: If you require SSL for the PowerShell virtual directory, you will render Remote PowerShell inoperable!

Once you’ve configured the redirection and SSL settings, open a command prompt and run iisreset. At this point you should be able to browse to http://localhost on the CAS server and get redirected to https://owa.customer.com/owa. These steps were tested on Windows Server 2008 R2. While they should be similar under Windows Server 2008, they may not be identical.

McAfee EPO and NLB Clusters

I rolled out McAfee agents to about ten servers yesterday and four of them wouldn’t show up properly in the console. I did the usual troubleshooting with this, played with cmdagent, and didn’t really get far. What I did eventually notice is that while two of my machines would show up as Managed in the console, the other two simply refused. The wierd behavior here is that if I went in cmdagent and resent all the properties of the missing servers, suddenly the other two would disappear from the console. I did a bit of research and it appears that McAfee generates the client identifier based on the machine’s MAC Address.

This piece of information was the ticket here. The machines in question are in two Windows Network Load Balancing (NLB) clusters. The high level summary of how NLB works is that it shares a MAC address amongst all the machines in the cluster. McAfee was using this MAC to generate agent identifiers and was coming up with duplicates in both clusters. I did a bit of research and McAfee has an article in their knowledge base as to how to deal with this with a workaround.

When you go to create the registry value described in the article, McAfee prevents it because they’re hooking things and making sure you don’t touch their part of the registry. This is implemented in the Access Protection Policy for VirusScan. If you look at the VirusScan log, you’ll see a line like this and the corresponding regedit error:

1/17/2010    12:26:02 PM    Blocked by Access Protection rule     DOMAIN\bdesmond    C:\Windows\regedit.exe    \REGISTRY\MACHINE\SOFTWARE\Wow6432Node\Network Associates\ePolicy Orchestrator\New Value #1    Common Standard Protection:Prevent modification of McAfee Common Management Agent files and settings    Action blocked : Create

image

To work around this, I edited my Access Protection policy and disabled blocking for the three “Prevent modification of McAfee files and settings” rules highlighted below:

image