Category Archives: 17309

An Open Letter to Microsoft Learning

Yesterday I was notified of a new video touting how Microsoft Learning is revamping its Exchange 2013 exams and certification requirements for Exchange 2013 SP1. As someone who has worked with Microsoft to help rewrite exams for Exchange 2010 SP1, I was interested to see what @MSLearning had to say. I was met with great disappointment when I was greeted with the following video (since removed, but still found on YouTube).






In response I tweeted on Twitter, “One more reason that customers need the MCM program. I weep for our future.




As a Microsoft Certified Master and someone who takes great pride in the 77 Microsoft certifications I hold, I take Microsoft certifications seriously. As a Microsoft Gold Partner ExtraTeam does, as well, and makes its mark in the professional services industry by hiring the the most highly certified consultants and engineers in the industry.



Judging by the feedback I received to my tweet, I know that other IT Pros share my sense of frustration and dismay about the direction of Microsoft Learning.



Veronica Wei Sopher of Microsoft Learning responded to my tweet, genuinely asking for my feedback – so here it is:



  • Take yourself seriously. “Sesame Street” style videos have no place in a professional certification program. As one person wrote, “The costumes? No names? This needs to feel more work related if the sound is muted.” How do you think this looks to hiring managers? I can’t imagine anything like this coming from the Cisco or CCISP certification programs.

  • Be respectful and show ownership. Many IT Pros, such as myself, have invested significant amounts of time preparing for, taking exams, and maintaining their Microsoft certifications. Many do it on their own time and with their own money. It’s embarrassing and insulting to all IT Pros to be associated with a program that makes fun of certifications and the process.

  • Have integrity. Like other MCM candidates, I spent 21 days in Redmond learning 24×7 about Exchange in the MCM program and I’d do it again in a heartbeat. It was one of the best learning experiences of my life. That’s why it was so disappointing when Microsoft Learning canceled the MCM program without any notice, even to the Exchange product group. When Tim Sneath canceled the program in September 2013, he told us that Microsoft Learning was looking into ways to revamp the program. It’s been nearly a year later and we have heard absolutely nothing. At this time, the highest level of certification that IT Pros can achieve is an MCSE, which is pretty much worthless due to cheating and brain dumps. There has to be a better top-tier certification for Microsoft products than what is available now.



Does your environment need an Exchange 2013 Edge Transport server?




I was asked to write an article for my friends at ENow Consulting, “Does your environment need an Exchange 2013 Edge Transport server?”  The standard consulting answer applies: “It depends.”



If you’re wondering if an Exchange Edge Transport server makes sense in your Exchange environment, I encourage you to head over to the ENow Consulting Blog to read the article.




SMTP Firewall Requirements for Exchange Online

Most of my Office 365 engagements are hybrid projects connecting Office 365 with Exchange on-premises, and most are with larger companies concerned with securing the hybrid deployment.



Exchange Online Protection servers send SMTP emails using a TLS connection usually to the hybrid or Edge Transport server to enable mail flow between cloud and on-prem users. Microsoft does not support any sort of SMTP gateway or appliance between EOP and the Edge or hybrid server. For this reason, customers normally have to open TCP port 25 on the firewall to the hybrid server from the Exchange Online Protection servers.








Companies can secure this SMTP traffic by configuring the perimeter firewall to allow inbound TCP 25 traffic only from Exchange Online Protection servers to the hybrid or Edge servers.



I’ve seen a number of articles that list the public IP addresses used by EOP to send SMTP emails to on-prem customers, but the one true list is maintained in the article, Exchange Online Protection IP Addresses. Currently, this article lists seven IPv4 blocks and one IPv6 block for SMTP delivery to on-prem:

  • 65.55.88.0/24
  • 207.46.51.64/26
  • 207.46.163.0/24
  • 213.199.154.0/24
  • 213.199.180.128/26
  • 216.32.180.0/24
  • 216.32.181.0/24
  • 2a01:111:f400:7c00::/54

Microsoft tries hard to not make changes to this list, but if they do they will update the article. It’s important for firewall admins to know that EOP does not use URLs for root domain routing (also known as Top-Level Domains, or TLDs). You must use the IP addresses listed in the article above.

Up until April 2014, Microsoft used many other IP addresses to send emails from Office 365 tenants to on-prem customers. This is because they maintain another set of IP addresses for something called the High Risk Delivery Pool, which is used to protect the production Exchange Online namespace from “spammy” senders. EOP no longer uses the High-Risk Delivery Pool when sending emails between the customer’s tenant and their on-prem servers.


It’s nice to know that we now have a single source to point to when configuring firewalls for Office 365.


Reporting Outlook Client Versions Using Log Parser

I’m doing a little cross-pollination today. Chris Lehr, one of my colleagues at ExtraTeam, worked up a Log Parser script that produces a report showing all the clients versions connecting to Exchange.  This is very helpful to show which clients are running Office versions in your organization that should be updated prior to migration.  Check out his blog post here.



Outlook Client Version Report

Clients will always get the best experience using the latest version of Office, currently Office 2013 SP1. The best practice is to always update your clients with the latest cumulative update prior to migration. this is especially true when you’re migrating to Office 365, since most updates pertain to Office 365, Exchange Online, and Exchange 2013 compatibility.



If you find that you need to upgrade clients to a new version of Office, I recommend that you install the x86 version of Office to provide the best compatibility with add-ons and third-party products. Some customers think they need to install Office x64 on Windows x64 operating systems, but that’s not the case. See 64-bit editions of Office 2013 for details on when it makes sense to install Office x64.



If you’re an Office 365 customer, I strongly recommend checking out using the Office 2013 ProPlus software deployment that’s most likely part of your Enterprise license. This version of Office 2013 can be installed on up to 5 PCs, iPads, tablets, etc. and is always up-to-date since it’s a cloud-managed service.




Fix for ASP.NET 4.0.30319.0 – 3005 Event message: An unhandled exception has occurred on Exchange 2013

I noticed after installing Windows Updates the following warning in the
Application Event log of all Exchange 2013 SP1 servers (abbreviated for
clarity):



Event 1309, ASP.NET 4.0.30319.0


Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Date:          5/3/2014 9:31:25 AM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      EX2013-1.contoso.com
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 5/3/2014 9:31:25 AM
Event time (UTC): 5/3/2014 4:31:25 PM
Event ID: 20e50da04e9745e1a73bf21fa1dbb509
Event sequence: 2
Event occurrence: 1
Event detail code: 0

Application information:
    Application domain: /LM/W3SVC/3/ROOT/owa-3-130436082719776031
    Trust level: Full
    Application Virtual Path: /owa
    Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\owa\
    Machine name: EX2013-1

Process information:
    Process ID: 6068
    Process name: w3wp.exe
    Account name: NT AUTHORITY\SYSTEM

Exception information:
    Exception type: MapiExceptionIllegalCrossServerConnection
    Exception message: MapiExceptionIllegalCrossServerConnection: Monitoring mailbox [] with application ID [Client=OWA] is not allowed to make cross-server calls from [EX2013-1.contoso.com] to [EX2013-2.contoso.com]
   at Microsoft.Mapi.CrossServerDiagnostics.BlockCrossServerCall(ExRpcConnectionInfo connectionInfo, String mailboxDescription)
   …
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore, Boolean unifiedSession)

Request information:
    Request URL: https://localhost:444/owa/proxylogon.owa
    Request path: /owa/proxylogon.owa
    User host address: ::1
    User: contoso\SM_ce56bab178eb42fda
    Is authenticated: True
    Authentication Type: Kerberos
    Thread account name: NT AUTHORITY\SYSTEM

Thread information:
    Thread ID: 15
    Thread account name: NT AUTHORITY\SYSTEM
    Is impersonating: False
    Stack trace:    at Microsoft.Mapi.CrossServerDiagnostics.BlockCrossServerCall(ExRpcConnectionInfo connectionInfo, String mailboxDescription)
   …
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore, Boolean unifiedSession)


The text highlighted above steered me toward the Exchange Health Monitoring
Mailboxes, so I ran Get-Mailbox -Monitoring and got the following results:

Name                      Alias                ServerName       ProhibitSendQuota
—-                      —–                ———-       —————–
HealthMailbox9a621ae8e… HealthMailbox9a62… ex2013-2         Unlimited
WARNING: The object contoso.com/Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailbox9a621ae8e6f341638c0c2161affa7645 has been corrupted, and it’s in an inconsistent
 state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.
WARNING: Database is mandatory on UserMailbox.
HealthMailbox1cd7a25f1… HealthMailbox1cd7… ex2013-2         Unlimited
HealthMailboxde79dfaa0… HealthMailboxde79… ex2013-2         Unlimited
WARNING: The object contoso.com/Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailboxde79dfaa09604ffd8578527c2d3ffab1 has been corrupted, and it’s in an inconsistent
 state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.
WARNING: Database is mandatory on UserMailbox.
HealthMailbox5e5bf093c… HealthMailbox5e5b… ex2013-2         Unlimited
HealthMailbox79f8d5d0e… HealthMailbox79f8… ex2013-2         Unlimited
WARNING: The object contoso.com/Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailbox79f8d5d0e02443d2a6acdc60bd0a026e has been corrupted, and it’s in an inconsistent
 state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.
WARNING: Database is mandatory on UserMailbox.
HealthMailbox693969aae… HealthMailbox6939… ex2013-2         Unlimited
HealthMailboxab01377ba… HealthMailboxab01… ex2013-2         Unlimited
WARNING: The object contoso.com/Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailboxab01377bae994825ba08d083552e196e has been corrupted, and it’s in an inconsistent
 state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.
WARNING: Database is mandatory on UserMailbox.
HealthMailbox7561b21db… HealthMailbox7561… ex2013-2         Unlimited
WARNING: The object contoso.com/Microsoft Exchange System Objects/Monitoring
Mailboxes/HealthMailbox7561b21db2c642778177f4ab0a2be350 has been corrupted, and it’s in an inconsistent
 state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.
WARNING: Database is mandatory on UserMailbox.
HealthMailboxb337fe270… HealthMailboxb337… ex2013-2         Unlimited
HealthMailboxfd8b0f99f… HealthMailboxfd8b… ex2013-2         Unlimited
HealthMailbox739cff7e6… HealthMailbox739c… ex2013-2         Unlimited
HealthMailboxd8337c5a1… HealthMailboxd833… ex2013-2         Unlimited
HealthMailboxa990ff65c… HealthMailboxa990… ex2013-1         Unlimited
HealthMailbox91e52135c… HealthMailbox91e5… ex2013-1         Unlimited
HealthMailboxf351943c2… HealthMailboxf351… ex2013-1         Unlimited
HealthMailbox23eb6e495… HealthMailbox23eb… ex2013-1         Unlimited
HealthMailbox1821ba284… HealthMailbox1821… ex2013-2         Unlimited

Besides the fact that some of the Health Mailboxes were missing the mandatory
Database parameter, there were far too many Health Mailboxes. The easy way to
correct this is to delete all the Exchange Health Mailboxes and recreate them.



Open Active Directory
Users and Computers and enable Advanced View. The Health Mailboxes are located under Microsoft Exchange System Objects \ Monitoring Mailboxes. Select all of the HealthMailbox objects, delete them, and replicate AD.



Microsoft Exchange System Objects \ Monitoring Mailboxes

Now that the Exchange Health Mailboxes are gone, restart the Microsoft
Exchange Health Manager
service on all Exchange 2013 servers to recreate the
necessary Exchange Health Mailboxes:



[PS] C:\>Restart-Service MSExchangeHM

Replicate AD again and you will see the new Exchange Health Mailboxes in the Exchange Management Shell and ADUC:

[PS] C:\>Get-Mailbox -Monitoring

Name Alias ServerName ProhibitSendQuota
—- —– ———- —————–
HealthMailbox7b3ef7a60… HealthMailbox7b3e… ex2013-1 Unlimited
HealthMailbox312d14677… HealthMailbox312d… ex2013-2 Unlimited
HealthMailbox1aec3204b… HealthMailbox1aec… ex2013-1 Unlimited
HealthMailboxa04a8b769… HealthMailboxa04a… ex2013-2 Unlimited
HealthMailbox04e954fc9… HealthMailbox04e9… ex2013-1 Unlimited
HealthMailboxd20957258… HealthMailboxd209… ex2013-2 Unlimited



Recreated Health Mailboxes

Two monitoring mailboxes are created for each mailbox database in your
organization: one for monitoring the health of site mailboxes and one for
monitoring the health of public folders. That should resolve the ASP.NET
4.0.30319.0 warnings.


Fix for Exchange ActiveSync Failures After Migration to Exchange 2013

There’s a bug in Exchange 2013 that causes Exchange ActiveSync to fail for newly migrated users from Exchange 2010. It only affects migrated users who already have a mobile device configured, not new users (i.e., test mailboxes). This issue was discussed in the Exchange Server forums back in August 2013.


The issue occurs because the IIS application pools on the CAS 2013 servers do not automatically detect that the mailbox has been moved to Exchange 2013. When the user’s mobile device connects to CAS 2013, CAS 2013 proxies the user back to CAS 2010 which responds with an error saying the mailbox is corrupt or missing. If you run an Exchange ActiveSync test using ExRCA you will see the X-CalculatedBETarget value reported by CAS2013 is still pointing to the Exchange 2010 server. The problem usually resolves itself in 1-8 hours, depending on the Exchange 2013 build.

The workaround is to manually recycle the MSExchangeAutodiscoverAppPool and MSExchangeSyncAppPool application pools in IIS on all CAS2013 servers.

I wrote a PowerShell script for this called Recycle-AppPools.ps1:

#Recycle-AppPools.ps1
#Jeff Guillet, MCM|MCSM|MVP|CISSP
#Use this script to recycle IIS Application Pools to overcome Exchange 2013 SP1 ActiveSync bug for migrated users

#Get all Exchange 2013 CAS servers
$CASServers = Get-ClientAccessServer | where {$_.WorkloadManagementPolicy -ne $null}

#Loop through each CAS2013 and recycle the IIS App Pools
foreach ($CAS in $CASServers) {
  Write-Host “Recycling App Pools on $CAS…”
  $appPool = Get-WmiObject -Authentication PacketPrivacy -Impersonation Impersonate -ComputerName $CAS -namespace “root/MicrosoftIISv2″ -class IIsApplicationPool | Where-Object {$_.Name -eq “W3SVC/AppPools/MSExchangeAutodiscoverAppPool” }
  $appPool.Recycle()
  $appPool = Get-WmiObject -Authentication PacketPrivacy -Impersonation Impersonate -ComputerName $CAS -namespace “root/MicrosoftIISv2″ -class IIsApplicationPool | Where-Object {$_.Name -eq “W3SVC/AppPools/MSExchangeSyncAppPool” }
  $appPool.Recycle()
}

You will need to run the script after an EAS user or batch of users have been migrated. There is no outage associated with recycling the app pools and it recycles very quickly. A fix is scheduled for Exchange 2013 CU5.



MEC 2014 is Right Around the Corner! Are You Registered?




I’m very much looking forward to the Microsoft Exchange Conference March 31-April 2 in Austin, TX. I hope you can join me there!



I’ll be moderating three MEC Unplugged interactive sessions this year and will be on the experts panel for the Exchange Deployment session.









Session Date/Time Room Speakers
Experts Unplugged: Architecture – Client Access and Connectivity  Tuesday, April 1 9:00AM – 10:15AM MR 17b Greg Taylor, Jeff Guillet, Jeff Mealiffe, Ross Smith IV, Venkat Ayyadevara
Experts Unplugged: Architecture – Transport and Hygiene Tuesday, April 1 10:45AM – 12:00PM MR 17b Brian Reid, Jeff Guillet, Khushru Irani, Ross Smith IV, Scott Landry, Wendy Wilkes
Experts Unplugged: Architecture – Transport and Hygiene (repeat session) Wednesday, April 2 8:30AM – 9:45AM MR 13ab Brian Reid, Jeff Guillet, Khushru Irani, Ross Smith IV, Scott Landry, Wendy Wilkes
Wednesday, April 2 8:30AM – 9:45AM MR 17b Brian Day, Greg Taylor, Jeff Guillet, Jeff Mealiffe, Ross Smith IV, Scott Schnoll



Click the sessions above to add them to your MEC schedule. If you haven’t registered for MEC yet, it’s not too late.



There will be no paper copy of the schedule this year. The schedule will be available via an HTML5 app that should work on all platforms, but I suggest you print a copy of your schedule or add it to your calendar before you arrive. Technology sometimes has a nasty way of not working when you need it.



Here’s a breakdown of the current attendee profile, based on registered attendees so far.






With 87% of the attendees from the US, it looks like Europe would definitely be served well by having its own MEC. That would better align with the global deployment of Exchange Server. I’d say the chances of seeing MEC in Europe would be pretty slim, though.


Troubleshooting TLS SMTP Connections to Exchange Online Protection

By default Office 365 uses Transport Layer Security (TLS) to send encrypted SMTP emails between Exchange Online and Exchange on-prem. This provides end-to-end encryption of emails between your on-prem Exchange Hybrid Server and Exchange Online Protection (EOP), just like they were the same organization.



TLS utilizes x.509 (SSL) certificates to encrypt the email in transport. Exchange Online and Exchange 2013’s implementation of TLS differs from previous implementations of TLS in several important ways:

  • The certificates used for TLS must be issued by a trusted third-party CA (Digicert, Godaddy, Verisign, etc.)
  • The CRL for the certification authority must be available. If Exchange or O365 can’t read the CRL it will not trust the certificate.
  • The FQDN that the Receive Connector provides in response to EHLO must match the subject name or a subject alternative name on the certificate. SAN certificates and wildcard certificates are both valid for TLS use. If you use a wildcard cert the FQDN used on the connector can be any name that is valid for the wildcard cert’s domain.
  • In previous versions of TLS any certificate could be used to encrypt SMTP traffic, even expired or self-signed certificates. This is not the case with Office 365 and Exchange 2013.
As previously mentioned, TLS is required by default for SMTP communication between Hybrid servers and Exchange Online Protection (EOP). When you run the Hybrid Configuration Wizard it will configure forced TLS for all Send and Receive Connectors. Note that this configuration is updated whenever you run the HCW, so if you reconfigure the connectors for opportunistic TLS or turn it off completely, the HCW will reconfigure them to use forced TLS again.

The following options are available for TLS encryption in Office 365:

  • Force TLS – Email sent across this connector must use TLS. If TLS is unavailable messages will queue until they are delivered or expired.
  • Opportunistic TLS - The connector tries to setup a TLS connection using the STARTTLS verb. If a TLS connection cannot be made the connector falls back to regular ESMTP or SMTP. This is functionally equivalent to turning TLS off.
Normally the use of TLS is configured on Receive or Inbound Connector. You do this to control how email is accepted by your domain and it can be easily configured using the Exchange Admin Console. 
Office 365’s Inbound Connector from Hybrid Server
Send Connectors can also be configured to require TLS. This is used to enforce that email is sent by this connector must only use TLS, and is configured in the shell using the RequireTLS property:

Outbound Connector TLS Settings to Office 365

Using this configuration if the corresponding Receive Connector does not offer or require TLS, messages will queue on the sending server until they are finally delivered or expire.

You can easily tell if a receiving SMTP server is configured to use TLS using Telnet. Install the Telnet Client feature, Telnet to the server using TCP port 25, and look for the STARTTLS verb after issuing the EHLO domain.com command. For example:

Telnet servername 25

If you see that the STARTTLS verb is missing, the server is not offering TLS. If your Send Connector is configured to require TLS, the messages will queue on the sending server with the error,
451 4.4.0 Primary target IP address responded with: “451 5.7.3 STARTTLS is required to send mail.”

There are several reasons that the STARTTLS verb might be missing:

  • The receiving server is not configured to Force TLS or use Opportunistic TLS.
  • The sending server’s IP is on an SMTP block list (aka SMTP blacklist or SMTP blocklist). Office 365 will not attempt to send TLS traffic with a server it can’t trust.
  • The receiving server is configured to only respond to SMTP (not ESMTP) commands. TLS is part of the ESMTP protocol.
  • Your firewall is doing some form of email inspection and is filtering “unsafe” verbs from the SMTP conversation. Some examples of firewalls that do this are:
Office 365 uses internal and external SMTP blocklists to protect the network. If the sending server is on one of these black lists the EOP servers will not offer TLS. You can test this by sending a non-TLS email to EOP using Telnet. See Brian Reid’s article, “Cannot Send Emails to Office 365 or Exchange Online Protection Using TLS“. The response from the EOP server will tell you which block list you’re on and how to request removal.

If the receiving server requires TLS, but the sending server is not configured to use a TLS certificate messages will queue on the sending server with the error,
451 4.4.0 Primary target IP address responded with: “451 5.7.3 Must issue a STARTTLS command first.”
The fix here is to configure the Send Connector to use TLS and a valid certificate using the Enable-ExchangeCertificate cmdlet, such as:
Enable-ExchangeCertificate -Thumbprint 5DC5902752816FD2FC51D5564C363F68D8F7FFC4 -Services SMTP
Make sure to use Get-ExchangeCertificate to get the correct certificate’s thumbprint for the command above.

Hopefully this information will help you understand TLS and will assist you with troubleshooting.


Chrome or Firefox will not accept credentials when logging into ADFS server

You may find that Google Chrome or FireFox 3.5+ keeps prompting for authentication when you are redirected to your ADFS 2.0 server.  This is also known to affect Fiddler.  See “AD FS 2.0: Continuously Prompted for Credentials While Using Fiddler Web Debugger” on TechNet.



This happens when Windows Authentication Extended Protection is enabled in IIS on either the ADFS proxy, ADFS back-end server, or both.



Here’s how to turn Extended Protection off:



  • Login to the ADFS proxy server and open the Internet Information Services (IIS) Management console.
  • Navigate to the adfs\ls virtual directory under the Default Web Site.


  • Double-click Authentication to open the authentication methods for the ADFS\LS directory.
  • Select Windows Authentication and then click Advanced Settings in the Actions pane.
  • Set Extended Protection to Off.
  • Make sure you do this for all your ADFS proxy and ADFS back-end servers.


Chrome or Firefox will not accept credentials when logging in using ADFS server

You may find that Google Chrome or FireFox 3.5+ keeps prompting for authentication when you are redirected to your ADFS 2.0 server.  This is also known to affect Fiddler.  See “AD FS 2.0: Continuously Prompted for Credentials While Using Fiddler Web Debugger” on TechNet.



This happens when Windows Authentication Extended Protection is enabled in IIS on either the ADFS proxy, ADFS back-end server, or both.



Here’s how to turn Extended Protection off:



  • Login to the ADFS proxy server and open the Internet Information Services (IIS) Management console.
  • Navigate to the adfs\ls virtual directory under the Default Web Site.



  • Double-click Authentication to open the authentication methods for the ADFS\LS directory.
  • Select Windows Authentication and then click Advanced Settings in the Actions pane.
  • Set Extended Protection to Off.
  • Make sure you do this for all your ADFS proxy and ADFS back-end servers.