Category Archives: 16750

Does your environment need an Exchange 2013 Edge Transport server?




I was asked to write an article for my friends at ENow Consulting, “Does your environment need an Exchange 2013 Edge Transport server?”  The standard consulting answer applies: “It depends.”



If you’re wondering if an Exchange Edge Transport server makes sense in your Exchange environment, I encourage you to head over to the ENow Consulting Blog to read the article.




SMTP Firewall Requirements for Exchange Online

Most of my Office 365 engagements are hybrid projects connecting Office 365 with Exchange on-premises, and most are with larger companies concerned with securing the hybrid deployment.



Exchange Online Protection servers send SMTP emails using a TLS connection usually to the hybrid or Edge Transport server to enable mail flow between cloud and on-prem users. Microsoft does not support any sort of SMTP gateway or appliance between EOP and the Edge or hybrid server. For this reason, customers normally have to open TCP port 25 on the firewall to the hybrid server from the Exchange Online Protection servers.








Companies can secure this SMTP traffic by configuring the perimeter firewall to allow inbound TCP 25 traffic only from Exchange Online Protection servers to the hybrid or Edge servers.



I’ve seen a number of articles that list the public IP addresses used by EOP to send SMTP emails to on-prem customers, but the one true list is maintained in the article, Exchange Online Protection IP Addresses. Currently, this article lists seven IPv4 blocks and one IPv6 block for SMTP delivery to on-prem:

  • 65.55.88.0/24
  • 207.46.51.64/26
  • 207.46.163.0/24
  • 213.199.154.0/24
  • 213.199.180.128/26
  • 216.32.180.0/24
  • 216.32.181.0/24
  • 2a01:111:f400:7c00::/54

Microsoft tries hard to not make changes to this list, but if they do they will update the article. It’s important for firewall admins to know that EOP does not use URLs for root domain routing (also known as Top-Level Domains, or TLDs). You must use the IP addresses listed in the article above.

Up until April 2014, Microsoft used many other IP addresses to send emails from Office 365 tenants to on-prem customers. This is because they maintain another set of IP addresses for something called the High Risk Delivery Pool, which is used to protect the production Exchange Online namespace from “spammy” senders. EOP no longer uses the High-Risk Delivery Pool when sending emails between the customer’s tenant and their on-prem servers.


It’s nice to know that we now have a single source to point to when configuring firewalls for Office 365.


Fix for Exchange ActiveSync Failures After Migration to Exchange 2013

There’s a bug in Exchange 2013 that causes Exchange ActiveSync to fail for newly migrated users from Exchange 2010. It only affects migrated users who already have a mobile device configured, not new users (i.e., test mailboxes). This issue was discussed in the Exchange Server forums back in August 2013.


The issue occurs because the IIS application pools on the CAS 2013 servers do not automatically detect that the mailbox has been moved to Exchange 2013. When the user’s mobile device connects to CAS 2013, CAS 2013 proxies the user back to CAS 2010 which responds with an error saying the mailbox is corrupt or missing. If you run an Exchange ActiveSync test using ExRCA you will see the X-CalculatedBETarget value reported by CAS2013 is still pointing to the Exchange 2010 server. The problem usually resolves itself in 1-8 hours, depending on the Exchange 2013 build.

The workaround is to manually recycle the MSExchangeAutodiscoverAppPool and MSExchangeSyncAppPool application pools in IIS on all CAS2013 servers.

I wrote a PowerShell script for this called Recycle-AppPools.ps1:

#Recycle-AppPools.ps1
#Jeff Guillet, MCM|MCSM|MVP|CISSP
#Use this script to recycle IIS Application Pools to overcome Exchange 2013 SP1 ActiveSync bug for migrated users

#Get all Exchange 2013 CAS servers
$CASServers = Get-ClientAccessServer | where {$_.WorkloadManagementPolicy -ne $null}

#Loop through each CAS2013 and recycle the IIS App Pools
foreach ($CAS in $CASServers) {
  Write-Host “Recycling App Pools on $CAS…”
  $appPool = Get-WmiObject -Authentication PacketPrivacy -Impersonation Impersonate -ComputerName $CAS -namespace “root/MicrosoftIISv2″ -class IIsApplicationPool | Where-Object {$_.Name -eq “W3SVC/AppPools/MSExchangeAutodiscoverAppPool” }
  $appPool.Recycle()
  $appPool = Get-WmiObject -Authentication PacketPrivacy -Impersonation Impersonate -ComputerName $CAS -namespace “root/MicrosoftIISv2″ -class IIsApplicationPool | Where-Object {$_.Name -eq “W3SVC/AppPools/MSExchangeSyncAppPool” }
  $appPool.Recycle()
}

You will need to run the script after an EAS user or batch of users have been migrated. There is no outage associated with recycling the app pools and it recycles very quickly. A fix is scheduled for Exchange 2013 CU5.



MEC 2014 is Right Around the Corner! Are You Registered?




I’m very much looking forward to the Microsoft Exchange Conference March 31-April 2 in Austin, TX. I hope you can join me there!



I’ll be moderating three MEC Unplugged interactive sessions this year and will be on the experts panel for the Exchange Deployment session.









Session Date/Time Room Speakers
Experts Unplugged: Architecture – Client Access and Connectivity  Tuesday, April 1 9:00AM – 10:15AM MR 17b Greg Taylor, Jeff Guillet, Jeff Mealiffe, Ross Smith IV, Venkat Ayyadevara
Experts Unplugged: Architecture – Transport and Hygiene Tuesday, April 1 10:45AM – 12:00PM MR 17b Brian Reid, Jeff Guillet, Khushru Irani, Ross Smith IV, Scott Landry, Wendy Wilkes
Experts Unplugged: Architecture – Transport and Hygiene (repeat session) Wednesday, April 2 8:30AM – 9:45AM MR 13ab Brian Reid, Jeff Guillet, Khushru Irani, Ross Smith IV, Scott Landry, Wendy Wilkes
Wednesday, April 2 8:30AM – 9:45AM MR 17b Brian Day, Greg Taylor, Jeff Guillet, Jeff Mealiffe, Ross Smith IV, Scott Schnoll



Click the sessions above to add them to your MEC schedule. If you haven’t registered for MEC yet, it’s not too late.



There will be no paper copy of the schedule this year. The schedule will be available via an HTML5 app that should work on all platforms, but I suggest you print a copy of your schedule or add it to your calendar before you arrive. Technology sometimes has a nasty way of not working when you need it.



Here’s a breakdown of the current attendee profile, based on registered attendees so far.






With 87% of the attendees from the US, it looks like Europe would definitely be served well by having its own MEC. That would better align with the global deployment of Exchange Server. I’d say the chances of seeing MEC in Europe would be pretty slim, though.


Troubleshooting TLS SMTP Connections to Exchange Online Protection

By default Office 365 uses Transport Layer Security (TLS) to send encrypted SMTP emails between Exchange Online and Exchange on-prem. This provides end-to-end encryption of emails between your on-prem Exchange Hybrid Server and Exchange Online Protection (EOP), just like they were the same organization.



TLS utilizes x.509 (SSL) certificates to encrypt the email in transport. Exchange Online and Exchange 2013’s implementation of TLS differs from previous implementations of TLS in several important ways:

  • The certificates used for TLS must be issued by a trusted third-party CA (Digicert, Godaddy, Verisign, etc.)
  • The CRL for the certification authority must be available. If Exchange or O365 can’t read the CRL it will not trust the certificate.
  • The FQDN that the Receive Connector provides in response to EHLO must match the subject name or a subject alternative name on the certificate. SAN certificates and wildcard certificates are both valid for TLS use. If you use a wildcard cert the FQDN used on the connector can be any name that is valid for the wildcard cert’s domain.
  • In previous versions of TLS any certificate could be used to encrypt SMTP traffic, even expired or self-signed certificates. This is not the case with Office 365 and Exchange 2013.
As previously mentioned, TLS is required by default for SMTP communication between Hybrid servers and Exchange Online Protection (EOP). When you run the Hybrid Configuration Wizard it will configure forced TLS for all Send and Receive Connectors. Note that this configuration is updated whenever you run the HCW, so if you reconfigure the connectors for opportunistic TLS or turn it off completely, the HCW will reconfigure them to use forced TLS again.

The following options are available for TLS encryption in Office 365:

  • Force TLS – Email sent across this connector must use TLS. If TLS is unavailable messages will queue until they are delivered or expired.
  • Opportunistic TLS - The connector tries to setup a TLS connection using the STARTTLS verb. If a TLS connection cannot be made the connector falls back to regular ESMTP or SMTP. This is functionally equivalent to turning TLS off.
Normally the use of TLS is configured on Receive or Inbound Connector. You do this to control how email is accepted by your domain and it can be easily configured using the Exchange Admin Console. 
Office 365’s Inbound Connector from Hybrid Server
Send Connectors can also be configured to require TLS. This is used to enforce that email is sent by this connector must only use TLS, and is configured in the shell using the RequireTLS property:

Outbound Connector TLS Settings to Office 365

Using this configuration if the corresponding Receive Connector does not offer or require TLS, messages will queue on the sending server until they are finally delivered or expire.

You can easily tell if a receiving SMTP server is configured to use TLS using Telnet. Install the Telnet Client feature, Telnet to the server using TCP port 25, and look for the STARTTLS verb after issuing the EHLO domain.com command. For example:

Telnet servername 25

If you see that the STARTTLS verb is missing, the server is not offering TLS. If your Send Connector is configured to require TLS, the messages will queue on the sending server with the error,
451 4.4.0 Primary target IP address responded with: “451 5.7.3 STARTTLS is required to send mail.”

There are several reasons that the STARTTLS verb might be missing:

  • The receiving server is not configured to Force TLS or use Opportunistic TLS.
  • The sending server’s IP is on an SMTP block list (aka SMTP blacklist or SMTP blocklist). Office 365 will not attempt to send TLS traffic with a server it can’t trust.
  • The receiving server is configured to only respond to SMTP (not ESMTP) commands. TLS is part of the ESMTP protocol.
  • Your firewall is doing some form of email inspection and is filtering “unsafe” verbs from the SMTP conversation. Some examples of firewalls that do this are:
Office 365 uses internal and external SMTP blocklists to protect the network. If the sending server is on one of these black lists the EOP servers will not offer TLS. You can test this by sending a non-TLS email to EOP using Telnet. See Brian Reid’s article, “Cannot Send Emails to Office 365 or Exchange Online Protection Using TLS“. The response from the EOP server will tell you which block list you’re on and how to request removal.

If the receiving server requires TLS, but the sending server is not configured to use a TLS certificate messages will queue on the sending server with the error,
451 4.4.0 Primary target IP address responded with: “451 5.7.3 Must issue a STARTTLS command first.”
The fix here is to configure the Send Connector to use TLS and a valid certificate using the Enable-ExchangeCertificate cmdlet, such as:
Enable-ExchangeCertificate -Thumbprint 5DC5902752816FD2FC51D5564C363F68D8F7FFC4 -Services SMTP
Make sure to use Get-ExchangeCertificate to get the correct certificate’s thumbprint for the command above.

Hopefully this information will help you understand TLS and will assist you with troubleshooting.


Chrome or Firefox will not accept credentials when logging in using ADFS server

You may find that Google Chrome or FireFox 3.5+ keeps prompting for authentication when you are redirected to your ADFS 2.0 server.  This is also known to affect Fiddler.  See “AD FS 2.0: Continuously Prompted for Credentials While Using Fiddler Web Debugger” on TechNet.



This happens when Windows Authentication Extended Protection is enabled in IIS on either the ADFS proxy, ADFS back-end server, or both.



Here’s how to turn Extended Protection off:



  • Login to the ADFS proxy server and open the Internet Information Services (IIS) Management console.
  • Navigate to the adfs\ls virtual directory under the Default Web Site.



  • Double-click Authentication to open the authentication methods for the ADFS\LS directory.
  • Select Windows Authentication and then click Advanced Settings in the Actions pane.
  • Set Extended Protection to Off.
  • Make sure you do this for all your ADFS proxy and ADFS back-end servers.


Chrome or Firefox will not accept credentials when logging into ADFS server

You may find that Google Chrome or FireFox 3.5+ keeps prompting for authentication when you are redirected to your ADFS 2.0 server.  This is also known to affect Fiddler.  See “AD FS 2.0: Continuously Prompted for Credentials While Using Fiddler Web Debugger” on TechNet.



This happens when Windows Authentication Extended Protection is enabled in IIS on either the ADFS proxy, ADFS back-end server, or both.



Here’s how to turn Extended Protection off:



  • Login to the ADFS proxy server and open the Internet Information Services (IIS) Management console.
  • Navigate to the adfs\ls virtual directory under the Default Web Site.


  • Double-click Authentication to open the authentication methods for the ADFS\LS directory.
  • Select Windows Authentication and then click Advanced Settings in the Actions pane.
  • Set Extended Protection to Off.
  • Make sure you do this for all your ADFS proxy and ADFS back-end servers.


Exchange 2013 Client Access Coexistence with Exchange 2010

The following guide explains how Exchange 2013 Client Access coexists with Exchange 2010 during a long-term migration.



Note: If you are migrating from Exchange 2007 please see my companion article.



Greg Taylor had a fabulous session on Microsoft Exchange Server 2013 Client Access Server Roleat TechEd 2013. I highly encourage you to watch it. For reference I’ve noted the time in the presentation that relates to each Exchange protocol below.  So, off we go…



The Key to Enlightenment


CAS2013 proxies everything.  Legacy namespace is not used.  Specific URLS are used if set, but if a client comes to the wrong place CAS2013 will proxy to just “make it work”.  Making it work is better than making it efficient. 




Autodiscover (8:40)

Internal and external DNS must be configured to point to Exchange 2013 CAS.

Client à CAS2013 (proxyà CAS2010



Outlook Anywhere (OA) (19:17)

CAS 2010 must be enabled for Outlook Anywhere. If OA2010 authentication is already set to NTLM you’re set, but if you’re using Basic authentication:
CAS2010 – OA Client authentication: Basic / OA IIS authentication: Must include NTLM

  • Get-OutlookAnywhere -Server <server> | Set-OutlookAnywhere -ExternalClientAuthenticationMethod Basic -InternalClientAuthenticationMethod Basic -IISAuthenticationMethods NTLM
CAS2013 - OA Client authentication: Basic / OA IIS authentication: Basic

  • Get-OutlookAnywhere -Server <server> | Set-OutlookAnywhere -ExternalClientAuthenticationMethod Basic -InternalClientAuthenticationMethod Basic -IISAuthenticationMethods Basic

Client à CAS2013 (proxyà CAS2010 à MBX2010



Outlook Web App (OWA) (23:00)

No legacy.contoso.com namespace used here. It’s pure proxy. CAS2010 SP3+ allows silent proxy even if FBA is on CAS2010.  2013 CU2+ provides SSO with multiple Internet-facing namespaces.

Client à CAS2013 (proxyà CAS2010 à MBX2010



ActiveSync (EAS) (31:30)

Client à CAS2013 (proxyà CAS2010 à MBX2010



Exchange Web Services (EWS) (35:30)

EWS settings always come from autodiscover.

Client à CAS2013 (proxyà CAS2010 à MBX2010



Exchange 2013 CU3 upgrade removes the ActiveSync virtual directory ExternalURL

Microsoft released Exchange Server 2013 Cumulative Update 3 on November 25, 2013.  CU3 includes fixes for many customer reported issues, minor product enhancements and previously released security bulletins.  A complete list of customer reported issues resolved in Exchange Server 2013 Cumulative Update 3 can be found in Knowledge Base Article KB 2892464.



Although it’s not listed in the Description of Cumulative Update 3 for Exchange Server 2013, one of the important things that Exchange 2013 CU3 fixes is OWA redirection for OWA 2010.  I discussed this in my article, OWA 2013 CU1 Redirection is Broken for Legacy Mailboxes.



Unfortunately, one important fix did not make it into CU3: Upgrading Exchange 2013 blanks out the ActiveSyncVirtualDirectory ExternalUrl property.  This only affects Exchange 2013 upgrades from RTM through CU2 – it does not affect new installations of Exchange 2013.



Exchange 2013 ActiveSyncVirtualDirectory URL values before CU3



Exchange 2013 ActiveSyncVirtualDirectory URL values after CU3



This problem was actually introduced back in the Exchange 2010 timeframe and has been present since Exchange 2013 CU1.  If you perform a RecoverServer operation in 2010, the operation should reconfigure the recovered server with the existing values stored in Active Directory.  At some point code was introduced that reset the URL values and security configuration of the OWA, ECP and ActiveSync to their default values (InternalURL uses the FQDN of the server and ExternalURL is blank).  Since Exchange 2013 servicing is a build-to-build upgrade it behaves pretty much like a RecoverServer operation and the problem becomes apparent.



The Exchange Team was able to address this behavior for the OWA and ECP virtual directories, but unfortunately did not address it for ActiveSync (it’s still being tracked).  Fixing the issue for OWA and ECP virtual directories was high priority since this affects users’ ability to access ECP and options from Outlook.



ActiveSync, however, only uses the external URL value once when a mobile device is first configured for email.  EAS devices do not periodically check Autodiscover and reconfigure themselves like Outlook does, so if the value is blank devices configured prior to the CU3 upgrade will still work.



This does affect new devices, however.  If the ExternalURL value for the ActiveSync is blank when a user tries to configure a new EAS mail client, the device has no idea how to connect to Exchange.  EAS will then prompt the user for the server name, which they may or may not know.



The fix is simple: After upgrading Exchange Server 2013 to CU3 reconfigure the ActiveSync virtual directory ExternalURL property using the following cmdlet:

Set-ActiveSyncVirtualDirectory -Server <servername> -ExternalUrl https://<external fqdn>/Microsoft-Server-ActiveSync

You may also need to reconfigure security settings for the ActiveSync virtual directory (i.e., if you use user certificates for authentication).  I recommend making note of your ActiveSync virtual directory URLs and security settings prior to the upgrade.



Hopefully, this will be completely addressed in Exchange 2013 SP1 due early 2014.



How to Share Free/Busy Information Without Using Federated Delegation

If you have a two-way forest trust in place between two organizations, such as during a merger, you can share calendar free/busy information between two Exchange organizations without having to configure federated delegation.



Federated delegation is usually the way to go since it offers more than just free/busy sharing.  With federation using the Microsoft Federated Gateway Server service you can share limited details between business partners.  In addition to free/busy information you can share meeting subjects, attendee, and location information.



Federated delegation works by brokering a trust between two different SMTP domains. The following diagram shows how this brokered trust works between contoso.com and fabrikam.com:



Federated delegation between contoso.com and fabrikam.com

To see more information about federated delegation see my articles, How to Configure Exchange 2010 SP1 Federation and Exchange Federated Free/Busy doesn’t work in one direction.



Because federated delegation relies on different SMTP domains, it won’t work during a merger/migration where both companies share the same SMTP domain (contoso.com).  The answer is to use the Add-AvailabilityAddressSpace cmdlet in both orgs.  



Be aware that the TechNet documentation for the Add-AvailabilityAddressSpace cmdlet is inaccurate/incomplete. It says this cmdlet is only supported on Exchange 2010 SP2 and Exchange 2010 SP3 when it’s actually supported on all versions of Exchange 2007 through Exchange 2013.  It also misses some crucial steps, which I’ve added below.



Here’s how to configure simple free/busy sharing information when both orgs share the same SMTP domain. For the examples below, the source domain is the one retrieving the free/busy info from the target domain. It is assumed that both domains already have a two-way forest trust configured and DNS forwarding is configured.

  • Run the following cmdlet in the target domain:
Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -User “sourcedomain\Exchange Servers”

  • Run the following two cmdlets in the source domain:
Add-AvailabilityAddressSpace -ForestName targetdomain.com -AccessMethod PerUserFB -UseServiceAccount $true
Export-AutodiscoverConfig -TargetForestDomainController dc.remotedomain.com -TargetForestCredential $Target -MultipleExchangeDeployments $true


After these steps are performed the users in the source domain can see free/busy information for users in the target domain.  Repeat these steps, switching the the source and target domains, to allow bidirectional free/busy look-ups.



Be aware that the source domain relies on Autodiscover and Exchange Web Services (EWS) in the target domain.  If Autodiscover is not configured properly (i.e., missing the autodiscover.targetdomain.com record in DNS) it will not work.  It also relies on the externalUrl property of the WebServicesVirtualDirectory in the target domain.  That must be set to a valid URL and the SSL certificate used by this EWS URL must be trusted by the source domain.