MEC 2014 is Right Around the Corner! Are You Registered?

I’m very much looking forward to the Microsoft Exchange Conference March 31-April 2 in Austin, TX. I hope you can join me there!

I’ll be moderating three MEC Unplugged interactive sessions this year and will be on the experts panel for the Exchange Deployment session.

Session Date/Time Room Speakers
Experts Unplugged: Architecture – Client Access and Connectivity  Tuesday, April 1 9:00AM – 10:15AM MR 17b Greg Taylor, Jeff Guillet, Jeff Mealiffe, Ross Smith IV, Venkat
Experts Unplugged: Architecture – Transport and Hygiene Tuesday, April 1 10:45AM – 12:00PM MR 17b Brian Reid, Jeff Guillet, Khushru Irani, Ross Smith IV, Scott Landry,
Wendy Wilkes
Unplugged: Architecture – Transport and Hygiene
(repeat session)
Wednesday, April 2 8:30AM – 9:45AM MR 13ab Brian Reid, Jeff Guillet, Khushru Irani, Ross Smith IV, Scott Landry,
Wendy Wilkes
Wednesday, April 2 8:30AM – 9:45AM MR 17b Brian Day, Greg Taylor, Jeff Guillet, Jeff Mealiffe, Ross Smith IV,
Scott Schnoll

Click the sessions above to add them to your MEC schedule. If you haven’t registered for MEC yet, it’s not too late.

There will be no paper copy of the schedule this year. The schedule will be available via an HTML5 app that should work on all platforms, but I suggest you print a copy of your schedule or add it to your calendar before you arrive. Technology sometimes has a nasty way of not working when you need it.

Here’s a breakdown of the current attendee profile, based on registered attendees so far.

With 87% of the attendees from the US, it looks like Europe would definitely be served well by having its own MEC. That would better align with the global deployment of Exchange Server. I’d say the chances of seeing MEC in Europe would be pretty slim, though.

Announcing the 7th Annual UC Roundtable at TechEd 2014, Houston!

I’m pleased to announce the 7th Annual UC Roundtable at Microsoft TechEd North America 2014 in Houston, TX!

The purpose of the UC Roundtable is to gather Exchange and Lync admins, MCMs, MVPs, Exchange product group members, architects, and experts for a free-flowing discussion about issues, questions, and experiences related to Exchange, Office 365, and Lync Server.  If you work with Exchange, Office 365, or Lync you need to be here!

The UC Roundtable will be held Wednesday, May 14, 2014 from 6:00-8:00PM CDT and will be within walking distance or a short cab ride from the TechEd hotels. A special thanks to my friends at F5 who will be hosting the event for the third year in a row!

Please RSVP to for event details and location. I will email you with the location details once they’re set.

Help spread the word and I hope you can join me!

Pending Messages in Exchange Online Will Not Be Delivered

There’s a pretty big unpublicized bug in Office 365 transport that you need to be aware of.
Messages from Exchange Online Protection (EOP) that cannot be delivered to an on-prem Exchange hybrid server due to a transport problem will be marked as Deferred or Pending.  These messages will NEVER be delivered, even after the transport issue is resolved.
The following example shows a message that was sent at 2:46PM UTC and the status shows as Pending. The last event for this message shows DEFER at 3:29PM UTC with the detail, “The last attempt to deliver the message encountered an error”.
Sample Message Trace from Office 365
For example, say the TLS
certificate expires on your hybrid server. Inbound messages from EOP to the hybrid server will queue because the Outbound Connector is using Forced TLS, but the certificate is invalid. If you resolve the problem by renewing the cert or reconfiguring the Office 365 Outbound Connector to use opportunistic TLS the problem is solved for new emails – they get delivered right away, but the Pending messages will
never get resubmitted and eventually expire (after 48 hours).
Incredibly, there currently is no way for Microsoft to resubmit these messages like there is for on-prem Exchange. After opening a high-priority case with Microsoft Online, the only “solution” they could give is to contact all the senders and ask them to resend their emails.

Troubleshooting TLS SMTP Connections to Exchange Online Protection

By default Office 365 uses Transport Layer Security (TLS) to send encrypted SMTP emails between Exchange Online and Exchange on-prem. This provides end-to-end encryption of emails between your on-prem Exchange Hybrid Server and Exchange Online Protection (EOP), just like they were the same organization.

TLS utilizes x.509 (SSL) certificates to encrypt the email in transport. Exchange Online and Exchange 2013’s implementation of TLS differs from previous implementations of TLS in several important ways:
  • The certificates used for TLS must be issued by a trusted third-party CA (Digicert, Godaddy, Verisign, etc.)
  • The CRL for the certification authority must be available. If Exchange or O365 can’t read the CRL it will not trust the certificate.
  • The FQDN that the Receive Connector provides in response to EHLO must match the subject name or a subject alternative name on the certificate. SAN certificates and wildcard certificates are both valid for TLS use. If you use a wildcard cert the FQDN used on the connector can be any name that is valid for the wildcard cert’s domain.
  • In previous versions of TLS any certificate could be used to encrypt SMTP traffic, even expired or self-signed certificates. This is not the case with Office 365 and Exchange 2013.
As previously mentioned, TLS is required by default for SMTP communication between Hybrid servers and Exchange Online Protection (EOP). When you run the Hybrid Configuration Wizard it will configure forced TLS for all Send and Receive Connectors. Note that this configuration is updated whenever you run the HCW, so if you reconfigure the connectors for opportunistic TLS or turn it off completely, the HCW will reconfigure them to use forced TLS again.

The following options are available for TLS encryption in Office 365:

  • Force TLS – Email sent across this connector must use TLS. If TLS is unavailable messages will queue until they are delivered or expired.
  • Opportunistic TLS – The connector tries to setup a TLS connection using the STARTTLS verb. If a TLS connection cannot be made the connector falls back to regular ESMTP or SMTP. This is functionally equivalent to turning TLS off.
Normally the use of TLS is configured on Receive or Inbound Connector. You do this to control how email is accepted by your domain and it can be easily configured using the Exchange Admin Console. 
Office 365’s Inbound Connector from Hybrid Server
Send Connectors can also be configured to require TLS. This is used to enforce that email is sent by this connector must only use TLS, and is configured in the shell using the RequireTLS property:
Outbound Connector TLS Settings to Office 365
Using this configuration if the corresponding Receive Connector does not offer or require TLS, messages will queue on the sending server until they are finally delivered or expire.

You can easily tell if a receiving SMTP server is configured to use TLS using Telnet. Install the Telnet Client feature, Telnet to the server using TCP port 25, and look for the STARTTLS verb after issuing the EHLO command. For example:

Telnet servername 25
If you see that the STARTTLS verb is missing, the server is not offering TLS. If your Send Connector is configured to require TLS, the messages will queue on the sending server with the error,

451 4.4.0 Primary target IP address responded with: “451 5.7.3 STARTTLS is required to send mail.”

There are several reasons that the STARTTLS verb might be missing:
  • The receiving server is not configured to Force TLS or use Opportunistic TLS.
  • The sending server’s IP is on an SMTP block list (aka SMTP blacklist or SMTP blocklist). Office 365 will not attempt to send TLS traffic with a server it can’t trust.
  • The receiving server is configured to only respond to SMTP (not ESMTP) commands. TLS is part of the ESMTP protocol.
  • Your firewall is doing some form of email inspection and is filtering “unsafe” verbs from the SMTP conversation. Some examples of firewalls that do this are:
Office 365 uses internal and external SMTP blocklists to protect the network. If the sending server is on one of these black lists the EOP servers will not offer TLS. You can test this by sending a non-TLS email to EOP using Telnet. See Brian Reid’s article, “Cannot Send Emails to Office 365 or Exchange Online Protection Using TLS“. The response from the EOP server will tell you which block list you’re on and how to request removal.

If the receiving server requires TLS, but the sending server is not configured to use a TLS certificate messages will queue on the sending server with the error,

451 4.4.0 Primary target IP address responded with: “451 5.7.3 Must issue a STARTTLS command first.”

The fix here is to configure the Send Connector to use TLS and a valid certificate using the Enable-ExchangeCertificate cmdlet, such as:

Enable-ExchangeCertificate -Thumbprint 5DC5902752816FD2FC51D5564C363F68D8F7FFC4 -Services SMTP

Make sure to use Get-ExchangeCertificate to get the correct certificate’s thumbprint for the command above.

Hopefully this information will help you understand TLS and will assist you with troubleshooting.