Updated 4/16/2014

Preamble

Virtualization is a valuable asset for many organizations, including cloud computing. However, there were some drawbacks that many administrators weren’t aware of when implementing a Hyper-V infrastructure.

For example, there are ramifications of cloning servers without Sysprepping the base image first. Sysprep generate a new SID (the unique Security Identifier each machine has) upon first-time boot up. There are also ramifications with the Virtual host time service, which provides time to the virtual guests, but if the guests are part of an AD infrastructure, or if the guests are DCs, then the host time synchronization service will cause problems with the default AD forest time hierarchy, due to Kerberos’ five minute skew tolerance. Easy enough, it’s recommended to disable time synchronization on the host to prevent this from occurring.

One of the more important ramifications, which we will discuss in the section, involves virtualized snapshots and domain controllers and using the Revert feature to roll back a virtual machine to a previous point in time using a previously saved snapshot. The ramifications can effectively make a DC useless.

In this blog, we’ll talk about:

• What is a Snapshot?
• What is the USN?
• What is a USN Rollback?
• Windows Server 2012 Snapshot Support
• Windows Server 2012 Cloning Support

What is a Snapshot?

Hyper-V provides the ability to create a point-in-time copy of a virtual guest. The point-in-time copy is called a snapshot. The snapshot can be used to “revert” the virtual guest back to the point-in-time the snapshot was created.

Snapshots are a convenient means to return a virtual machine to a previous state, such as to return to a state prior to installing an application that is no longer behaving properly.

What is the USN?

The USN, or Update Sequence Number, is the basis of how Active Directory Replication works.

The USN is a value stored with each attribute that changes by either a local change, or a replicated change from a partner domain controller. Each domain controller keeps track of its own changes, and other domain controllers in the infrastructure are aware of all other domain controller USN value.

Active Directory replication relies on Update Sequence Numbers (USNs) on each domain controller. The USN acts as a counter. Each DC’s USN value is unique to a domain controller. The replication system is designed with this restriction in mind.

When an inbound replication partner domain controller sees its partner has a higher USN value for any attribute, a replication pull request is made to replicate the changes to the partner.

Active Directory Replication does not depend on or use time displacement or a time stamp to determine what changes need to be propagated. Time based propagation as some directory services use, are based on a time stamp with the “last writer wins” rule, however this can pose a problem if the clock were to be rolled back.

A time stamp is used in Active Directory, but it’s only used to determine and resolve a conflict when an attribute has been modified at two different DCs simultaneously. In this case, the DC receiving the update will use one of three values to resolve a conflict:

1. The Version number that is incremented on an attribute by the original writer
2. The originating time of the original writer
3. The originating DSA value, which is the GUID of the domain controller (found in ADSI Edit and in the _msdcs.contoso.com DNS zone).

And because these USN counters are local to each DC, it ensures and is determined that the USN to be reliable by replication partner DCs, because the local DC keeps track of all its own changes.

The USN can never “run backward” (decrease in value). If it does, replication partner DCs will recognize the decreased value, and determine it as an inconsistency, and will remove the DC from its own replica set. This is called a USN Rollback. Although they can be repaired, in many cases, it’s easier and more time efficient to simply force remove the DC with the USN Rollback, and re-promote it back into the domain.

You can use Ldp.exe or ADSI Edit to read the current USN, which is the highestCommittedUsn attribute that can be found on the RootDSE object properties for the domain controller.

Up-to-Dateness, High-Watermark, Propagation Dampening, InvocationID

Replication takes into account specific values and follows a pre-defined algorithm to insure replication consistency among domain controllers to reduce or eliminate divergence, such as the following:

Up-To-Dateness vector

• This is a value that the destination domain controller maintains for tracking the originating updates that are received from all source domain controllers.
  • This value helps the source DC filter irrelevant attributes (and entire objects if all attributes are filtered) on the basis of the relationships between all sources of originating updates and a single destination.
  • To see the Up-to-datenes vector value, run the repadmin /showvector command.

High-watermark

• This is a value that the destination domain controller maintains to keep track of the most recent change that it has received from a specific source domain controller for an object in a specific directory partition.
  • This value prevents irrelevant objects from being considered by the source domain controller with respect to a single destination.
  • To see the value of the High-watermark, run repadmin /showreps /verbose and look for each line that starts “USNs:”. The high-watermark USN is the number that is followed by “/OU”.

Propagation Dampening

Fault tolerance is helpful by installing multiple DCs, and provides multiple replication paths between them to reduce latency; however, you might expect the same replication change to be replicated in an endless loop. The Up-to-dateness vector eliminates this possibility along with the InvocationID. The InvocationID of a domain controller and its USN combined provides a unique identifier in the forest associated with every write-transaction performed on each domain controller.

Replication example

To understand the consequences of snapshots prior to Windows Server 2012 requires a brief explanation and basic understanding of how Active Directory replication works.

Scenario: Single Domain, single AD Site, three DCs. DC-A, DC-B, and DC-C, all are replication partners between each other.

Replication Steps

• DC-A updates a password. The USN is set to 3.
• DC-B detects a USN change on DC-A
• DC-B requests the change from DC-A
• DC-B sends its high-watermark and up-to-dateness vector to DC-A

—–> DC-A looks at the high-watermark and up-to-dateness vector values, and the object that was changed, (the password attribute).
—–> DC-A sees that the originating DSA for the password change is DC-A (itself).
—–> DC-A reads the up-to-dateness vector from DC-B and finds that DC-B is guaranteed to be Up-To-Date from the change from DC-A (itself), but has a USN value of 2.
—–> DC-A sees that the originating USN is 3 on that password attribute.

• Based on the fact 3 is greater than 2, DC-A sends the changed password to DC-B.

Summary

In summary, propagation dampening occurs if DC-B already received the changed password from DC-C, which received it from DC-A, therefore, DC-B will not request the changed password from DC-A.

Additional reading, and summarized from:
Tracking Updates (Active Directory Replication)
http://technet.microsoft.com/en-us/library/cc961798.aspx

 

Pre-Windows 2012 Virtualized DC recommendations

• Do not take snapshots or revert back to a snapshot of a domain controller virtual machine.
• Do not copy the domain controller VHD file.
• Do not export the virtual machine that is running a domain controller.
• Do not restore a domain controller or attempt to roll back the contents of an Active Directory database by any other means than a supported backup solution.

 

Undetected USN Rollback

From: Running Domain Controllers in Hyper-V
http://technet.microsoft.com/en-us/library/d2cae85b-41ac-497f-8cd1-5fbaa6740ffe(v=ws.10)#usn_and_usn_rollback

 

Detected USN Rollback

From: Introduction to Active Directory Domain Services (AD DS) Virtualization (Level 100)
http://technet.microsoft.com/en-us/library/hh831734.aspx

 

Repairing USN Rollbacks

To repair a USN Rollback may be difficult. You can use the replication monitoring and diagnostic tools to determine the extent of the damage. If severe where the USN Rollback is undetected, such as when the VHD file attached to a different virtual host is copied and run on another virtual host, which will make it extremely difficult to determine the cause due to duplicate DC SID numbers, besides the rollback, or if the USN on a restored DC has increased past the last USN that the other domain controller has received. In this case, the USN values of the originating DC are different than what the replication partner believes they should be.

The easiest way to repair a USN rollback is to force remove the domain controller that was reverted, run a metadata cleanup to remove the domain controller’s reference from the AD database, and re-promote it.

Reverting back to a snapshot can cause ramifications with other types of services. For one, you must keep in mind of the secure channel that is used by Active Directory members to communicate to the domain. The secured channel uses a password that gets renewed every seven days. For example, if you revert the machine back prior to the point with a previous password, it may no longer be able to communicate. To repair such a scenario, you can reset the machine account, or disjoin it then rejoin it back to the domain. For servers, such as a Microsoft Exchange server, the implications can be much deeper. Besides the secured channel, users will lose any emails that were received between the current time and snapshot time.

 

Windows Server 2012 Snapshot Support Prevents USN Rollbacks

Until the introduction of Windows Server 2012, cloning, snapshotting, or copying, are unsupported. The only supported method to repair a DC is to potentially either using Windows Backup, or a third party backup that supports non-Authoritative or Authoritative restores, or simply force demote and rebuild the DC from scratch and promote it back into the domain. Otherwise, as we’ve discussed, snapshots and cloning have serious ramifications that can result in USN rollbacks or lingering objects, just to name a few.

Windows Server 2012 now supports DC cloning and snapshot restore of domain controllers. The requirements to support the new feature are:

• Hypervisor that supports VM-GenerationID. Window Server 2012 Hyper-V supports VM-GenerationID. If using a third party Hypervisor, check with the vendor if their latest version supports this feature.
• The source virtual domain controller must be running Windows Server 2012.
• A Windows Server 2012 PDC Emulator FSMO Role must be running and available for the cloned DC.

 

How does the VM-GenerationID work?

When you promote a domain controller in a supported Hypervisor, AD DS stores the VM-GenerationID (msDS-GenerationID attribute) in the DC’s computer object in the Ad database. This attribute will now be tracked by a Windows driver in the virtual machine.

If you revert to a snapshot, the driver looks at the current VM-GenerationID value and compares it to the value in the AD database on its computer object. The comparison also occurs each time a DC is rebooted.

If the VM-GenerationID are different:

• The InvocationID is reset
• The RID pool is deleted
• The new value is updated in the AD database, thus preventing any possibility of the USN values to be re-used.
• A non-authoritative SYSVOL synchronization occurs to safely restore and re-initialize SYSVOL (to prevent a JRNL-WRAP error).
• Each time a DC is rebooted, the value is compared, and if they are different, this rule and action applies.
• These actions also safeguards shutdown virtual DCs.

If the VM-GenerationID are the same:

• The snapshot and transaction is committed.

 

Windows Server 2012 Cloning

In Windows Server 2012, administrators no longer need to use Sysprep to clone a machine, promote it to a domain controller, then complete any additional tasks such as Windows Updates, or install organization standard applications. After the first domain controller is freshly installed from scratch or using Sysprep in a domain, Administrators can now safely deploy cloned domain controllers by simply copying an existing virtual domain controller.

This feature is domain specific. A domain must have at least one DC installed that can be copied. You still want to properly configure DNS settings, validate each DC’s health, replication status, and run the Active Directory Best Practice Analyzer after each Dc deployment.

This feature provides the following advantageous and benefits:

• Rapid DC deployment
• Quick restores
• Optimize private cloud deployments
• Rapid DC provisioning to quickly meet increased capacity needs

What if I Don’t Want the VM-Generation ID Mechanism to Kick In?

Perhaps there’s a time when you don’t want this protection, such as if you are trying to clone your environment to a lab. If you follow the rules, the VM-Generation ID will protect the USN and probably not give you what you want, and worse, if the DCs you’re trying to clone are having trouble replicating SYSVOL, you have more problems to deal with.

One way around it to prevent the VM-Generation ID to kick in at the hypervisor level is to shut down the VMs, and simply do a flat file copy to another hypervisor, then create a new VM from using the existing files.That should help the attribute mechanism from kicking in. More info on this and other thoughts:

Cases where VM-GenerationID doesn’t help make Active Directory virtualization-safe -Part 1
http://blogs.dirteam.com/blogs/sanderberkouwer/archive/2013/08/28/cases-where-vm-generationid-doesn-t-help-make-active-directory-virtualization-safe-part-1.aspx

Why Windows Server 2012 AD VM-Generation ID functionality is not an alias for Active Directory anti-USN Rollback functionality
http://blog.joeware.net/2013/02/20/2675/

*

Additional Reading:

Tracking Updates (USN & Active Directory Replication)
http://technet.microsoft.com/en-us/library/cc961798.aspx

Running Domain Controllers in Hyper-V
http://technet.microsoft.com/en-us/library/d2cae85b-41ac-497f-8cd1-5fbaa6740ffe(v=ws.10)#usn_and_usn_rollback

How to detect and recover from a USN rollback in Windows Server 2003, Windows Server 2008, and Windows Server 2008 R2
http://support.microsoft.com/kb/875495

Steps for deploying a clone virtualized domain controller
http://technet.microsoft.com/en-us/library/hh831734.aspx#steps_deploy_vdc

Virtual Domain Controller Cloning in Windows Server 2012
http://blogs.technet.com/b/askpfeplat/archive/2012/10/01/virtual-domain-controller-cloning-in-windows-server-2012.aspx

By Ace Fekay

MCT, MVP, MCSE 2012/Cloud, MCITP EA, MCTS Windows 2008/R2, Exchange 2007 & 2010, Exchange 2010 Enterprise Administrator, MCSE 2003/2000, MCSA Messaging 2003
  Microsoft Certified Trainer
  Microsoft MVP: Directory Services
  Active Directory, Exchange and Windows Infrastructure Engineer

Comments are welcomed.