There’s a pretty big unpublicized bug in Office 365 transport that you need to be aware of.
Messages from Exchange Online Protection (EOP) that cannot be delivered to an on-prem Exchange hybrid server due to a transport problem will be marked as Deferred or Pending. These messages will NEVER be delivered, even after the transport issue is resolved.
The following example shows a message that was sent at 2:46PM UTC and the status shows as Pending. The last event for this message shows DEFER at 3:29PM UTC with the detail, “The last attempt to deliver the message encountered an error”.
|Sample Message Trace from Office 365|
For example, say the TLS
certificate expires on your hybrid server. Inbound messages from EOP to the hybrid server will queue because the Outbound Connector is using Forced TLS, but the certificate is invalid. If you resolve the problem by renewing the cert or reconfiguring the Office 365 Outbound Connector to use opportunistic TLS the problem is solved for new emails – they get delivered right away, but the Pending messages will
never get resubmitted and eventually expire (after 48 hours).
Incredibly, there currently is no way for Microsoft to resubmit these messages like there is for on-prem Exchange. After opening a high-priority case with Microsoft Online, the only “solution” they could give is to contact all the senders and ask them to resend their emails.
By default Office 365 uses Transport Layer Security (TLS) to send encrypted SMTP emails between Exchange Online and Exchange on-prem. This provides end-to-end encryption of emails between your on-prem Exchange Hybrid Server and Exchange Online Protection (EOP), just like they were the same organization.
TLS utilizes x.509 (SSL) certificates to encrypt the email in transport. Exchange Online and Exchange 2013’s implementation of TLS differs from previous implementations of TLS in several important ways:
- The certificates used for TLS must be issued by a trusted third-party CA (Digicert, Godaddy, Verisign, etc.)
- The CRL for the certification authority must be available. If Exchange or O365 can’t read the CRL it will not trust the certificate.
- The FQDN that the Receive Connector provides in response to EHLO must match the subject name or a subject alternative name on the certificate. SAN certificates and wildcard certificates are both valid for TLS use. If you use a wildcard cert the FQDN used on the connector can be any name that is valid for the wildcard cert’s domain.
- In previous versions of TLS any certificate could be used to encrypt SMTP traffic, even expired or self-signed certificates. This is not the case with Office 365 and Exchange 2013.
As previously mentioned, TLS is required by default for SMTP communication between Hybrid servers and Exchange Online Protection (EOP). When you run the Hybrid Configuration Wizard it will configure forced TLS for all Send and Receive Connectors. Note that this configuration is updated whenever you run the HCW, so if you reconfigure the connectors for opportunistic TLS or turn it off completely, the HCW will reconfigure them to use forced TLS again.
The following options are available for TLS encryption in Office 365:
- Force TLS – Email sent across this connector must use TLS. If TLS is unavailable messages will queue until they are delivered or expired.
- Opportunistic TLS - The connector tries to setup a TLS connection using the STARTTLS verb. If a TLS connection cannot be made the connector falls back to regular ESMTP or SMTP. This is functionally equivalent to turning TLS off.
Normally the use of TLS is configured on Receive or Inbound Connector. You do this to control how email is accepted by your domain and it can be easily configured using the Exchange Admin Console.
|Office 365’s Inbound Connector from Hybrid Server|
Send Connectors can also be configured to require TLS. This is used to enforce that email is sent by this connector must only use TLS, and is configured in the shell using the RequireTLS property:
|Outbound Connector TLS Settings to Office 365|
Using this configuration if the corresponding Receive Connector does not offer or require TLS, messages will queue on the sending server until they are finally delivered or expire.
You can easily tell if a receiving SMTP server is configured to use TLS using Telnet. Install the Telnet Client feature, Telnet to the server using TCP port 25, and look for the STARTTLS verb after issuing the EHLO domain.com command. For example:
|Telnet servername 25|
If you see that the STARTTLS verb is missing, the server is not offering TLS. If your Send Connector is configured to require TLS, the messages will queue on the sending server with the error,
451 4.4.0 Primary target IP address responded with: “451 5.7.3 STARTTLS is required to send mail.”
There are several reasons that the STARTTLS verb might be missing:
- The receiving server is not configured to Force TLS or use Opportunistic TLS.
- The sending server’s IP is on an SMTP block list (aka SMTP blacklist or SMTP blocklist). Office 365 will not attempt to send TLS traffic with a server it can’t trust.
- The receiving server is configured to only respond to SMTP (not ESMTP) commands. TLS is part of the ESMTP protocol.
- Your firewall is doing some form of email inspection and is filtering “unsafe” verbs from the SMTP conversation. Some examples of firewalls that do this are:
Office 365 uses internal and external SMTP blocklists to protect the network. If the sending server is on one of these black lists the EOP servers will not offer TLS. You can test this by sending a non-TLS email to EOP using Telnet. See Brian Reid’s article, “Cannot Send Emails to Office 365 or Exchange Online Protection Using TLS
“. The response from the EOP server will tell you which block list you’re on and how to request removal.
If the receiving server requires TLS, but the sending server is not configured to use a TLS certificate messages will queue on the sending server with the error,
451 4.4.0 Primary target IP address responded with: “451 5.7.3 Must issue a STARTTLS command first.”
The fix here is to configure the Send Connector to use TLS and a valid certificate using the Enable-ExchangeCertificate cmdlet, such as:
Enable-ExchangeCertificate -Thumbprint 5DC5902752816FD2FC51D5564C363F68D8F7FFC4 -Services SMTP
Make sure to use Get-ExchangeCertificate to get the correct certificate’s thumbprint for the command above.
Hopefully this information will help you understand TLS and will assist you with troubleshooting.
Most organizations have internal application servers and appliances that send emails to users or groups. Examples include copier/scanners and application servers, such as backup servers that notify admins of a completed or failed backup job.
If the organization has Exchange on-prem you would normally configure an internal relay send connector in Exchange and configure the internal resources to send emails to Exchange. But what do you do when you’ve migrated all your mailboxes to Office 365 and have decommissioned your Exchange servers?
The solution is to install an IIS SMTP relay server in your internal network, configure it to accept email from specific IP addresses, and forward emails to Office 365. You can also configure the SMTP relay for external domains, if necessary.
Here’s how to do it:
- Install the SMTP Server feature and its dependencies to a new or existing Windows server. This will be your relay server and your firewall needs to allow it to send SMTP traffic (TCP port 25) outbound to the Internet. I typically use the DirSync server, if there is one.
|Adding the SMTP Server feature and its dependencies to Windows Server 2012|
- Open Internet Information Services (IIS) 6.0 Manager to configure the SMTP relay.
- Configure the properties of [SMTP Virtual Server #1] as follows:
- On the Access tab:
- Authentication: Only Anonymous access is checked.
- Relay: Only the list below. Add IP addresses or ranges of servers allowed to relay.
- Note – It’s important to only allow IP addresses you trust to relay through this server. Any IP address you enter here will be allowed to send emails on behalf of your domain.
- On the Messages tab:
- Adjust message size limits. The default message size limit is 2048 KB (2 MB). You may want to change it to 10240 KB (10 MB) or more to allow for larger messages from copier/scanners, etc.
- On the Delivery Tab:
- Outbound Security: Anonymous access only and no TLS encryption.
- Outbound Connections: Port 25
- Advanced: Leave the Smart Host field blank
- Add new remote domains:
- Right-click Domains > New > Domain and add the domain(s) hosted in Exchange Online.
- If the relay server is allowed to relay emails to other external domains add a new *.com remote domain. Repeat for *.org, *.net, etc. as necessary.
|Add Office 365 and other remote domains if required for external relay|
- For properties of each domain hosted in Exchange Online:
- Check Allow incoming mail to be relayed to this domain
- Forward all mail to this smart host: smtp.office365.com
- Outbound Security: Check Anonymous access and TLS encryption
- For properties of all other remote domains (if any):
- Check Allow incoming mail to be relayed to this domain
- Outbound Security: Check Anonymous access and do not check TLS encryption.
- Restart IIS. Be aware that whenever you restart IIS, the SMTP virtual server usually stays stopped – start it.
- The SMTP Server feature can be added to any Windows 2003 or better server. I usually use the DirSync server if there is one.
- Unlike Exchange, TLS for IIS 6 SMTP servers is not opportunistic. If the virtual server or a remote domain is configured to use TLS email will not be sent if the remote domain does not support TLS. Office 365 offers TLS, so we can use it.
- The configuration above allows the IIS 6 SMTP server to send emails to the Internet for the remote domains configured, so you should add the public NAT IP address for this server to your existing SPF record to prevent non-delivery. Use http://whatismyip.com from the SMTP server to determine the NAT IP address.
- Monitor the %systemdrive%\Inetpub\mailroot\Queue folder to ensure that emails are being delivered.
- If emails are not being delivered to Office 365 users, test sending email via Telnet. The IP address may be blocked by an Exchange Online Protection (EOP) blocklist and you will see that response from EOP. If so, send a delist request from your Office 365 admin account to email@example.com letting them know the IP address that should be delisted. In my experience it takes up to 36 hours for Microsoft to delist it.
- If emails are not being delivered to external domains, ensure that you have a remote domain type (*.com, *.eu, etc.) configured for those email addresses.
- You can enable logging in the properties of the SMTP virtual server for further troubleshooting. Use the NCSA Common Log File Format. IIS does not automatically groom or delete logs like Exchange does, so turn logging off when you’re done troubleshooting.
- The best practice is to create an A record in internal DNS for smtp.yourdomain.com using the SMTP relay server’s IP, and configure all application servers and appliances to use that FQDN for email forwarding. That makes it easier to update in the future.