Don’t Deploy Exchange 2013 CU6 If You’re a Hybrid Customer

I have confirmed With Microsoft that there are significant bugs in Exchange 2013 Cumulative Update 6 for hybrid customers.

Update #1: Microsoft just published a new article, Exchange Server 2013 databases unexpectedly fail over in a co-existence environment with Exchange Server 2007, which describes a different issue where Exchange 2013 databases unexpectedly fail over between the nodes of database availability groups. A hotfix is available for this issue, but you have to call Microsoft Support to get it.

Update #2: Microsoft just published another new article, Exchange Online mailboxes cannot be managed by using EAC after you deploy Exchange Server 2013 CU6, which provides a script that fixes the problems described in this article. Thankfully, you do not need to contact Microsoft Support to obtain the script, but you do need to configure PowerShell script execution to run it and you should know script resets IIS without prompting.  Run “Set-ExecutionPolicy -ExecutionPolicy unrestricted” to allow the script to run.

Hybrid deployments are used to bridge the gap between Exchange on-premises and Office 365. An Exchange hybrid server is used as the on-prem MRS endpoint for mailbox moves to Office 365, provides rich coexistence (free/busy sharing), and provides encrypted TLS mail flow between on-prem and Office 365.

Both Exchange 2010 and Exchange 2013 support hybrid servers. If the on-prem environment is Exchange 2010, the existing Exchange 2010 Hub/CAS servers can be used as hybrid servers, or new Exchange 2013 servers can be deployed. Exchange 2007 customers must deploy at least one new hybrid server and they usually deploy Exchange 2013.

Microsoft has maintained that customers will always be able to manage their hybrid environments from on-prem. Hybrid servers are supposed to bridge the administrative gap, providing a single pane of glass through which customers can manage both on-prem and Exchange Online environments.

That was until Exchange 2013 CU6…

With CU6, admins can no longer use the Exchange Admin Center (EAC) to create new Office 365 mailboxes, move mailboxes to Exchange Online, or create In-Place Archive mailboxes. Admins either need to use the Exchange Management Shell (EMS) or logon to the Office 365 Portal to perform these actions. In addition, when you click the Office 365 tab it normally takes you to the Office 365 signon portal so you can manage your Office 365 tenant  Instead, it opens a new website for the Office 365 marketing page. These are huge problems for most hybrid customers and there’s no mention of this anywhere in the CU6 release notes.

Here’s the experience in Exchange 2013 CU5:

CU5 – Create New Office 365 Mailbox

CU5 – Move Mailbox to Exchange Online

CU5 – Create In-Place Archive Online

Exchange 2013 CU6 hybrid customers are greeted with an entirely different experience:
CU6 – Admins Can Only Create On-Prem Mailboxes

CU6 – Admins Can Only Move Mailboxes to Another On-Prem Mailbox

CU6 – Admins Can Only Create On-Prem Archive Mailboxes

And here’s what Admins see when they click the Office 365 tab in the EAC:

CU6 – Office 365 Tab

I expect Microsoft to publish an article soon regarding these bugs, but with a long Labor Day weekend ahead of us I wouldn’t expect anything sooner that Tuesday. I do expect that CU7 will correct these bugs. In the meantime, I recommend that hybrid customers do not deploy CU6 at this time.  If you’ve already deployed CU6 in your environment, there’s no way to role back.

What do you think Microsoft should do? Pull CU6? Release an Interim Update, like they did for the CU5 hybrid bug? Leave your comment below.

Source: Expta

Thycotic Secret Server Product Review

I don’t normally do product reviews, but I like to share when I find something that works well.

I’ve been using Thycotic Secret Server for a while now to store my personal account information, passwords and account notes. It acts as a secure vault for for this important information. Prior to this, I’m ashamed to say, I was using the same username and password for most of my accounts. Obviously this is a terrible practice, especially in this day an age where banks, stores, and websites are frequently under attack for this information.

The Heartbleed Bug in OpenSSL brought this to the forefront for me. I knew I had to change all my passwords with new complex passwords, but the challenge of trying to remember all those passwords was an impossible task. I tested several different password management solutions, but none of them worked as well and as trouble free as Secret Server.

The following are the list of requirements I needed in a password management program:

  • Easy to use
  • Available remotely
  • Automatic complex password generation
  • Automatic login to password protected websites
  • Must work in the browsers I use (Internet Explorer and Chrome)
  • Must work with my iOS devices (iPhone and iPad)

Secret Server is just one of Thycotic’s security products aimed at securing your personal and private data. Thycotic offers a free* Express Edition of Secret Server for private use, and this is what I’m running. OK, technically it’s not “free” – it costs $10 per year, but Thycotic donates this to charity. Not only is this super cheap compared to other password management solutions, it also shows what a nice bunch these Thycotic folks are. Other editions have additional features and capabilities, such as the ability to change network passwords remotely, service account management, and provide high availability. I should also mention that all versions of Secret Server (including Express Edition) include full online support!

I installed Secret Server Express Edition on a dedicated Windows Server 2012 R2 web server, but you can also install it on an existing web server. You will need to install the IIS role and features, the .NET Framework 4.5.1, and Microsoft SQL Server 2012 Express. After that, the installation is a simple 5-step process and you can manage your passwords (secrets) right away. The comprehensive Secret Server Installation Guide walks you through the entire process, including prerequisites.

Once installed, you can access Secret Server through the IIS website you created. To add a new secret, select the Secret Template dropdown box in the upper right corner. The template you select contains all the relevant fields for the secret. I use the Web Password template for most of my secrets. This template allows me to use the Web Password Filler (described below).

Once a Web Password secret has been saved with the logon URL, username, and password, it’s easy to have Secret Server log you in to the website with the unique complex password. Simply add the Web Password Filler applet to the Favorites on your web browsers:

Then click the Web Password Filler favorite when you want to logon to the website. You will need to login to the Secret Server if you aren’t already, then Secret Server will automatically log you on to the website.  for example, here’s the automatic logon for Amazon:
Thycotic also has a free Secret Server app on the Apple App Store so you can access your secrets and passwords from iOS devices. It doesn’t offer the same auto sign-in feature, but it does provide easy access to launch logon URLs and copy complex passwords.

There are many other features that Secret Server provides, but I honestly haven’t had a need to use them myself. Some of these advanced features include:

  • Roles-based access controls
  • Full auditing and reports
  • Email notifications
If you’re looking for a full featured password management solution I encourage you give Secret Server a try. They offer a 30-day free trial.

Source: Expta

Fix for MSExchange Mailbox Replication EventID 1121 Error Every Minute

I found that an error was being reported every 60 seconds from the MSExchange Mailbox Replication service with eventID 1121 on an Exchange 2013 CU5 server.

Log Name:      Application
Source:        MSExchange Mailbox Replication
Date:          8/15/2014 12:01:15 PM
Event ID:      1121
Task Category: Request
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SACEXCH01.Domain.local
The Microsoft Exchange Mailbox Replication service was unable to process a request due to an unexpected error.
Request GUID: ‘b451acde-8d08-4a9a-a248-6bc4ca144aa2’
Database GUID: ‘ed87ca06-6ce2-448a-9a3b-2c9984b067a5’
Error: Database ‘aad284ae-7777-4896-93a5-cbc5e479841c’ doesn’t exist.
Stack trace:
   at Microsoft.Exchange.MailboxReplicationService.MapiUtils.FindServerForMdb(Guid mdbGuid, String dcName, NetworkCredential cred, FindServerFlags flags)
   at Microsoft.Exchange.MailboxReplicationService.MoveJob.ReserveLocalForestResources(ReservationContext reservation)
   at Microsoft.Exchange.MailboxReplicationService.MoveJob.AttemptToPick(MapiStore systemMailbox)
   at Microsoft.Exchange.MailboxReplicationService.SystemMailboxJobs.<>c__DisplayClassc.<ProcessJobsInBatches>b__6()
   at Microsoft.Exchange.MailboxReplicationService.CommonUtils.ProcessKnownExceptionsWithoutTracing(Action actionDelegate, FailureDelegate processFailure).

There are no open mailbox move requests, export requests, import requests, or migration batches.  I set diagnostic logging to expert, but nothing more than this single event every minute to nearly the second.
ed87ca06-6ce2-448a-9a3b-2c9984b067a5 resolves to an active database on this server.
aad284ae-7777-4896-93a5-cbc5e479841c does not resolve to any database in the org (obviously).
The fix is to remove the move request manually with the following cmdlet:

Remove-MoveRequest -MoveRequestQueue “ed87ca06-6ce2-448a-9a3b-2c9984b067a5” -MailboxGuid “b451acde-8d08-4a9a-a248-6bc4ca144aa2”

Immediately, the MSExchange Mailbox Replication event 1121 errors stopped.

Source: Expta