Category Archives: 13964

SMTP Firewall Requirements for Exchange Online

Most of my Office 365 engagements are hybrid projects connecting Office 365 with Exchange on-premises, and most are with larger companies concerned with securing the hybrid deployment.



Exchange Online Protection servers send SMTP emails using a TLS connection usually to the hybrid or Edge Transport server to enable mail flow between cloud and on-prem users. Microsoft does not support any sort of SMTP gateway or appliance between EOP and the Edge or hybrid server. For this reason, customers normally have to open TCP port 25 on the firewall to the hybrid server from the Exchange Online Protection servers.








Companies can secure this SMTP traffic by configuring the perimeter firewall to allow inbound TCP 25 traffic only from Exchange Online Protection servers to the hybrid or Edge servers.



I’ve seen a number of articles that list the public IP addresses used by EOP to send SMTP emails to on-prem customers, but the one true list is maintained in the article, Exchange Online Protection IP Addresses. Currently, this article lists seven IPv4 blocks and one IPv6 block for SMTP delivery to on-prem:

  • 65.55.88.0/24
  • 207.46.51.64/26
  • 207.46.163.0/24
  • 213.199.154.0/24
  • 213.199.180.128/26
  • 216.32.180.0/24
  • 216.32.181.0/24
  • 2a01:111:f400:7c00::/54

Microsoft tries hard to not make changes to this list, but if they do they will update the article. It’s important for firewall admins to know that EOP does not use URLs for root domain routing (also known as Top-Level Domains, or TLDs). You must use the IP addresses listed in the article above.

Up until April 2014, Microsoft used many other IP addresses to send emails from Office 365 tenants to on-prem customers. This is because they maintain another set of IP addresses for something called the High Risk Delivery Pool, which is used to protect the production Exchange Online namespace from “spammy” senders. EOP no longer uses the High-Risk Delivery Pool when sending emails between the customer’s tenant and their on-prem servers.


It’s nice to know that we now have a single source to point to when configuring firewalls for Office 365.


The ‘Heartbleed’ Security Flaw – Are You Affected?


(CNN) — A major online security vulnerability dubbed “Heartbleed” could put your personal information at risk, including passwords, credit card information and e-mails.

Heartbleed is a flaw in OpenSSL, an open-source encryption technology that is used by an estimated two-thirds of Web servers. It is behind many HTTPS sites that collect personal or financial information. These sites are typically indicated by a lock icon in the browser to let site visitors know the information they’re sending online is hidden from prying eyes.

Cybercriminals could exploit the bug to access visitors’ personal data as well as a site’s cryptographic keys, which can be used to impersonate that site and collect even more information.

You can use the Heartbleed Test website (http://filippo.io/Heartbleed/) to test your external websites and external-facing web appliances to see if they are vulnerable. I encourage you to make a quick test of your systems ASAP.



Exchange 2013 Health Check Monitors and Journaling

Exchange 2013 includes built-in health monitors that monitor the health of system resources.  Microsoft calls this new process “Managed Availability”.



The Exchange 2013 Server Health and Performance topic on TechNet says,
“Exchange 2013 introduces the concept of managed availability. Managed availability runs on every Exchange 2013 server. It’s made up of two processes, the Exchange Health Manager Service (MSExchangeHMHost.exe) and the Exchange Health Manager Worker process (MSExchangeHMWorker.exe), and the following asynchronous components:
  • Probe engine   The probe engine takes measurements on the server.
  • Monitoring probe engine   The monitoring probe engine stores the business logic about what constitutes a healthy state. It functions like a pattern recognition engine, looking for patterns and measurements that differ from a healthy state, and then evaluating whether a component or feature is unhealthy.
  • Responder engine   When the responder engine is alerted about an unhealthy component, its first action is to try to recover that component. Managed availability enables multi-stage recovery actions. The first attempt may be to restart the application pool, the second attempt may be to restart the corresponding service, and the third attempt may be to restart the server. And, the final attempt may be to put the server offline, so that it no longer accepts traffic. If all of these actions fail, an alert is sent to the help desk.”

When you install Exchange 2013 it automatically creates several HealthMailbox<guid> objects in Active Directory used by the managed availability service.  There are two health mailboxes that are created for a single mailbox database, one for mailboxes, and one for Public Folders (if deployed).  These hidden mailbox objects can be viewed from EMS by running the following command:

Get-Mailbox -Monitor

Exchange 2013 managed availability uses these HealthMailbox<guid> objects to send emails through Exchange to verify mail flow every 5 minutes.  This causes problems if you’re doing organization-wide journaling in the RTM version of Exchange 2013.  The org-wide Journal Rule will journal all these health probe emails, polluting the journal with thousands of useless messages.

Examples of these journaled health monitor messages are:



Sender: HealthMailboxac09e6fd942f4b40aba364cf2bf34e6b@contoso.com
Subject: MBTSubmission/StoreDriverSubmission/00000047-0000-0000-0000-0000b7145037-MapiSubmitLAMProbe
Message-Id: <2f33c299d6594de9b2ade02dee24e0c9@EX1.contoso.com>
To: HealthMailboxac09e6fd942f4b40aba364cf2bf34e6b@contoso.com

Sender: HealthMailboxc8a6d67ca286431abdf771e24a58093e@contoso.com
Subject: Client submission probe
Message-Id: <70701396-16e9-49ed-9888-11e39257d03e@EX1.contoso.com>
Bcc: HealthMailboxc8a6d67ca286431abdf771e24a58093e@contoso.com

Sender: inboundproxy@inboundproxy.com
Subject: Inbound proxy probe
Message-Id: <20d4e3a7-34a4-4b59-9773-9e1b51328b5a@EX1.contoso.com>
Recipient: HealthMailboxc8a6d67ca286431abdf771e24a58093e@contoso.com



Exchange 2013 Journal Mailbox filled with HealthMailbox* reports


This issue is supposed to be fixed in the first half of 2013, but if you can’t wait that long (who could blame you), here’s a workaround:

  • Add “Ignore” as the value of the ExtensionCustomAttribute1 attribute on each HealthMailbox* object in Active Directory using the following command from the Exchange Management Shell:
Get-Mailbox -Monitoring | Set-Mailbox -ExtensionCustomAttribute1 ‘Ignore’
  • Use the Exchange Management Shell to create a new Dynamic Distribution Group using the following two commands:
New-DynamicDistributionGroup -Name ‘Journaled Users’ -Alias JournaledUsers -RecipientFilter {((((CustomAttribute1 -ne ‘Ignore’) -and (RecipientType -eq ‘UserMailbox’))) -and (-not(Name -like ‘SystemMailbox{*’)) -and (-not(Name -like ‘CAS_{*’)) -and (-not(RecipientTypeDetailsValue -eq ‘MailboxPlan’)) -and (-not(RecipientTypeDetailsValue -eq ‘DiscoveryMailbox’)) -and (-not(RecipientTypeDetailsValue -eq ‘PublicFolderMailbox’)) -and (-not(RecipientTypeDetailsValue -eq ‘ArbitrationMailbox’)))}
Set-DynamicDistributionGroup -HiddenFromAddressListsEnabled $true
  • The commands above create an Exchange Dynamic Distribution Group called Journaled Users that contains all email enabled objects where the ExtensionCustomAttribute1 doesn’t equal Ignore.  It then hides the Dynamic Distribution Group from the Exchange address lists.  Note that users will not see this DDG in the list of groups they are members of.
  • Create a new journal mailbox to hold the journal reports and hide it from Exchange address lists.  In this example, I call it Journal Mailbox.
  • Lastly, create a new Journal Rule that journals all emails for the Journaled Users DDG to a journaling mailbox called Journal All using the following command from EMS:
New-JournalRule -Name ‘Journal All’ -JournalEmailAddress ‘journalmailbox@domain.com’ -Scope ‘Global’ -Enabled $true -Recipient ‘JournaledUsers@domain.com’

It’s important that you don’t update the Dynamic Distribution Group using the Exchange Management Console.  Doing so will update the DDG to a “precanned” RecipientFilter and the HealthMailbox* mailboxes will be journaled.



Fix for DCOM 10009 Errors in Exchange 2010 SP1

You may notice DistributedCOM 10009 errors in the Windows Server 2008 R2 System Event Log whenever you run any of the following Exchange 2010 SP1 cmdlets:

  • Get-OWAVirtualDirectory
  • Get-WebServicesDirectory
  • Get-ActiveSyncVirtualDirectory




The DCOM 10009 error reads as follows:



Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          7/1/2011 10:16:11 AM
Event ID:      10009
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CAS01.domain.com
Description:
DCOM was unable to communicate with the computer CAS02.domain.com using any of the configured protocols.
This happens because of an security context error when invoking an RPC call to the remote CAS server.  The fix is to direct the RPC Runtime to ignore delegation failures.  This can be done by configuring the registry on both the source and target machines, but is more easily done using Group Policy.



To configure Ignore Delegation Failures manually:

  • Run REGEDIT on the source computer
  • Navigate to HKLM\Software\Policies\Microsoft\Windows NT\Rpc
  • Create a new DWORD value called IgnoreDelegationFailure with the value of 1
  • Restart the computer
  • Repeat for each Exchange 2010 SP1 Client Access Server

 To configure this setting using Group Policy:

  • Open the Group Policy Management Console
  • Edit the Group Policy Object (GPO) that applies to the Exchange 2010 SP1 servers.  I usually use the Default Domain Policy.
  • Navigate to Computer Configuration > Policies > Administrative Templates > System > Remote Procedure Call
  • Double-click Ignore Delegation Failure.
  • Enable the policy and set the Ignoring Delegation Failure setting to ON.
  • Restart the Exchange 2010 SP1 Client Access Servers

This DCOM 10009 error does not seem to affect Windows Server 2008 servers, only Windows Server 2008 R2.

TechEd 2011: Day 3 Review and Random Photos

Today was a very busy day, as opposed to the other days that were just busy.  :)

I started with the session, “Exchange Server 2010 High Availability Management and Operations” with Scott Scnoll.  GREAT session with clear concise explanations of the cmdlets and scripts written for Exchange 2010 about HA and DR.  All of the content was new this year.  Be sure to view it at TechEd online.  The session is now available for download.



Next, I took in another security session, “Rethinking Cyber Threats: Expert Panel” with Laura Chappell, Marcus Murray, and Paula Januszkiewicz.  Topics included security, government regulations, IPv6 threats, and “How do you fix stupid”.  I was able to ask Marcus if Mark Russinovich detected the keylogger Marcus installed on his laptop at the MVP Summit using SysInternals tools, or if he just reimaged.  The answer was, “That’s under NDA. I want to come back next year.”



I then went to my first Lync session, “Troubleshoot Microsoft Lync 2010″.  All the Lync troubleshooting tools were covered except Snooper, for some reason.  What I found interesting was when he showed how to use the monitoring server to get metrics on call and video performance.



I had a TechEd focus group to end my day and then went back to the hotel to get ready for the real UC Roundtable.



[pictures later]

TechEd 2011: Day 1 Review and Random Photos

The Georgia World Congress Center

Whew, TechEd Day 1 is in the bag!  And speaking of which, here’s what the TechEd bag looks like.

TechEd 2011 Bag
Yesterday was a day to get registered, meet up with other people, and get bearings on the conference center.  After that, I was able to take in a game at Turner Field where the Braves beat the Phillies 3-2.


The evening ended with an epic Meet-n-Greet party with The Krewe at STATS bar in Atlanta.  A terrific time was had by all.  We had about 150 people show up, including old and new members of The Krewe.  It was a great time!

The Krewe Meet-N-Greet Party
Day 1 of TechEd Atlanta started with the keynote address at 9AM with Robert Wahbe, Corporate Vice President of Server and Tools, and Jason Zander, Corporate Vice President of Visual Studio.  As usual, the first hour was devoted to IT Pros and the second for developers.  And as expected, we heard how wonderful life is going to be when we move into the cloud.  As one person tweeted about 15 minutes into the keynote, “Drinking game is cancelled. If I took a drink every time keynote speaker said ‘cloud,’ I’d be dead by now.”

We followed up the keynote with a brief visit to the vendor area and a not so brief trip to the food area (aka, Tennessee, because it’s so far away from everything).  As we entered the food area, the wait staff started applauding and cheering us to have a great lunch.  It was like being in a parade.  Very surreal.

My first session was with Laura Chappell on “Wiretapping 101: Catching Evidence on the Network.”  It was a good session and was highly technical and entertaining.  Laura is great, but she needs some Ritalin.

I followed that up with my first Exchange session, “What’s New in Microsoft Exchange Server 2010 SP2: Featuring GAL Segmentation.”  We learned that SP2 will be released in the second half of this calendar year.  It will have over 500 bug fixes and include some knew features which require schema changes.  OWA Mini will be like a completely rewritten OMA for text-based browsers.  GAL segmentation allows you to have more than one Global Address List in the organization.  Users assigned the GAL policy 1 will only see users in that policy, and users assigned GAL policy 2 will only see their users.  There are quite a few caveats about this feature, like it doesn’t work for Mac clients.  It’s still early in the development cycle, so we’ll have to see how it pans out.

They will also have a fix for CAS redirection when a user signs into OWA from a less optimal site.  Rather than logging in and getting a link that says to use a different CAS, and then having to log in again on that CAS, the first CAS will automatically pass credentials to the second CAS.

From 6-9PM was the vendor reception party, where there was mass consumption of food and drink while collecting large amounts of SWAG.  Personally, I wasn’t too impressed with the food (bland/cold) and there wasn’t much in the way of SWAG.  No biggie, that’s not what I’m here for.  It was good hanging out with my friends.

Tomorrow I’ll be up bright and early, attending mostly Exchange and Lync sessions.

Here are some random photos from the day:

Welcome to TechEd 2011

The Alumni Lounge

The vendor area is being setup on Saturday

The $5 cell phone charger.  The free one is on the left.

Centennial Olympic Park

Me at Turner Field to watch the Braves

Our TechEd Krewe buttons

Me and Scott Ladewig at our special Blogger Hub area in the Connect Zone

Protect Your Windows Computer from Fraudulent Certificates

Today it was revealed that a serious security breach occurred at Comodo, a trusted certificate provider.  The breach appears to have come from Iran and several “high value certificates” were obtained.



These X.509 certificates include:

  • login.live.com
  • mail.google.com
  • www.google.com
  • login.yahoo.com (3 certificates)
  • login.skype.com
  • addons.mozilla.org
  • “Global Trustee”
To protect your Windows computer (PC or server) from trusting these high value certificates, download and install KB2524375 Microsoft Security Advisory: Fraudulent Digital Certificates could allow spoofing from Microsoft as soon as possible.  The installation takes only a minute and does not require a restart.



KB2524375 updates both the Computer’s and User’s Untrusted Certificates list to include the compromised certificates.



Here’s what the list looks like before the update:





And here’s what it looks like after the update:




Please take a minute to update your computers now.  This update is also being pushed out through Windows Update as I write this.

Disabling a User in AD Does Not Disable the User In Lync

It’s quite common for companies to disable user accounts in Active Directory, rather than delete them, when a user leaves the company.  This allows other IT staff and managers to access that user’s data and email after they are gone.



However, disabling a user account in Active Directory does not immediately disable the user from using Lync.  This is due to the way that Lync performs authentication and, depending on several factors, could result in a disabled user accessing Lync for up to nearly 6 months!  Obviously, this is important to understand since you don’t want disabled users to access internal resources or make Enterprise Voice calls.



The purpose of this article is to explain how and why this happens and how to successfully disable a Lync user’s account immediately without having to delete the user account from AD.



Lync Server 2010 uses several methods of authentication: Kerberos, NTLM, and certificate based.  Kerberos is the default authentication method and successful authentication results in the client receiving a Kerberos ticket that’s good for 10 hours.  Kerberos is used when users are accessing Lync Server while on the domain.  NTLM is used for authentication from other locations, such as the Internet for remote access using Lync Edge servers.




If the user authenticates using one of these two methods and selects the Save my password check box (shown above), the Lync server will generate an X.509 certificate for the user.  Lync will publish the certificate to Lync RTC database and distribute it, along with the private key, to the personal certificate store to the user on the local computer.  The certificate expires 180 days from the publication date and is used for further authentication for that user from that computer.  An example OCS signed certificate from the user’s Personal certificate store is shown below:





Certificate authentication is convenient and speeds up the sign-in process significantly, but it means that Lync doesn’t check the AD user account to see if it’s disabled.  If a disabled user signs into Lync using certificate authentication, they will still have access to all Lync features including IM, web conferencing and Enterprise Voice until the certificate expires.



The certificate(s) used by a Lync user can be viewed from the Lync Management Shell using the Get-CsClientCertificate cmdlet.  For example,



Get-CsClientCertificate sip:username@domain.com
will display all the certificates the certificates stored in the rtc database for that user. If the user has run Lync from three different computers, there will be three certificates listed for the user, as shown below:





Remote users with a valid client certificate can continue to sign in and access Lync until their certificate expires, regardless of whether their account is disabled or not.

You can revoke a certificate using the Revoke-CsClientCertificate cmdlet in the Lync Management Shell, but this will not affect users who are currently signed into Lync.  For domain computers, the user will be able to use Lync until their Kerberos ticket expires (up to 10 hours).  Remote users using certificate authentication will remain signed in until they sign out, the Lync server is restarted, or their certificate expires (up to 180 days).

To prevent a user (enabled or disabled) from using Lync, you must disable their Lync account using the Lync Control Panel or the Lync Management Shell, as shown below:



 




To disable the Lync user account using the Management Shell, run the following cmdlet:

Disable-CsUser sip:user@domain.com
Note that it may still take a few minutes for a signed-in user to become disconnected, however they will be unable to access any Lync features, such as new IM, web conferencing, or Enterprise Voice calls immediately.  If they happen to be in an IM session or web conference when their Lync account is disabled, they can continue until they disconnect.  Likewise, if they are in a voice call when their Lync account is disabled, the call will continue until the call ends.  The Lync client for the disabled user will display the following:

 



Thanks to Tom Pacyk for sharing this with me while he was at Microsoft Certified Master: Lync  Server training.

Adding users to local security groups using Group Policy

You may find that you need to add users to one or more local groups, such as Power Users or Administrators, on their computer.  While you can do this fairly easily on a case by case basis, it’s a lot more difficult to do in a large distributed environment.  This can be accomplished much easier using the Restricted Groups GPO setting in Group Policy.

The Restricted Group setting allows you to configure membership in groups within Active Directory or in the local security accounts manager (SAM) of domain-joined computers. 

In this example, we will add all domain users to the local computers’ Power Users group for all computers in the domain.
  • Open the Group Policy Management Console
  • Edit the Default Domain Policy
  • Navigate to Computer Configuration > Policies > Windows Settings > Security Settings > Restricted Groups
  • Right-click Restricted Groups and select Add Group…
  • The trick to adding a local group is to just type in the group name.  Do not browse to find the Power Users group, because this will resolve to the domain’s Power Users group.  Type Power Users, as shown below, and click OK.

  • Another window will pop-up to let you configure the properties of the Power Users Restricted Group.  For Members of this group, click Add.
  • Click the Browse button and browse for the group in Active Directory that you want to add to the local Power Users group.  In this example, use Domain Users and click OK, as shown below.

  • Close the GPO Editor and the Group Policy Management Console
Wait a sufficient amount of time to allow the GPO to replicate throughout all the domain controllers in the domain, then restart the computers where the policy applies.  This is required because the GPO affects the Computer Policy which applies when the computer starts up.

When the policy is processed, the computer will attempt to resolve the Power Users name that you typed to a local group first, then a domain group if no local match is found.

You can do the same process above for any other OU to scope the GPO to a specific set of computers.  If you want to add users to the local Administrators group, simply type that name instead of Power Users.

How to fix MSExchangeTransport Event ID 12014 on Edge and Hub Transport servers

By default, Exchange 2007 and 2010 attempt to use Transport Layer Security (TLS) for all SMTP traffic.  TLS uses a certificate on the receiving server to encrypt SMTP traffic between SMTP servers, similar to the way a certificate on the CAS server is used to secure OWA traffic.  If TLS cannot be negotiated, SMTP will usually fallback to non-encrypted SMTP.

In order for a server to send SMTP email via TLS:
  1. The receiving server must have an Exchange certificate in the computer’s local Personal store.
  2. The SMTP service must be assigned to use this certificate.
  3. The FQDN used in the Receive Connector must match either the Common Name or one of the Subject Alternative Names (if they exist) on the SMTP certificate.
If any one of these requirements is not met, you will see the following error in the application log of the Edge Transport server:

Log Name:      Application
Source:        MSExchangeTransport
Date:          9/28/2010 9:35:58 AM
Event ID:      12014
Task Category: TransportService
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mailgate
Description:
Microsoft Exchange could not find a certificate that contains the domain name mail1.expta.com in the personal store on the local computer. Therefore, it is unable to support the STARTTLS SMTP verb for the connector Default internal receive connector MAILGATE with a FQDN parameter of mail.expta.com. If the connector’s FQDN is not specified, the computer’s FQDN is used. Verify the connector configuration and the installed certificates to make sure that there is a certificate with a domain name for that FQDN. If this certificate exists, run Enable-ExchangeCertificate -Services SMTP to make sure that the Microsoft Exchange Transport service has access to the certificate key.
When you see this error on Edge Transport servers you have to examine the error description to determine where the mismatch occurs.  In the example above, the connector in error is the “Default internal receive connector MAILGATE“, which is the receive connector that exists on the Edge server itself.  If the connector in error is on the “EdgeSync – Inbound to domain” connector, the mismatch is on the Hub Transport server’s receive connector.

You can fix this by reconfiguring the offending connector to use the Common Name or Subject Alternative Name used on the Exchange certificate.  You can find this value by viewing the certificate from the Certificates MMC, as shown below:
To reconfigure the Edge Server’s Receive Connector:
  • On the Edge server, open the Exchange Management Console.
  • Navigate to Microsoft Exchange > EdgeTransport.
  • Click the Receive Connectors tab to view the existing connectors.
  • Double-click the Default internal receive connector SERVER connector to view its properties.
  • In the Specify the FQDN this connector will provide in response to HELO or EHLO field, enter the certificate’s Common Name (for example, ex1.expta.com) as shown below, and click OK.

To reconfigure the Hub Transport’s Receive Connector:
  • On the CAS, open the Exchange Management Console.
  • Navigate to Microsoft Exchange > Microsoft Exchange On-Premises > Organization Configuration > Hub Transport.
  • Click the Send Connectors tab to view the existing Send Connectors.
  • Double-click the EdgeSync – Inbound to domain connector to view its properties.
  • In the Specify the FQDN this connector will provide in response to HELO or EHLO field enter the certificate’s Common Name (for example, ex1.expta.com) as shown above, and click OK.