Windows Server 2012 AD Cloning, Snapshot Support & Preventing USN Rollbacks

Updated 4/16/2014


Virtualization is a valuable asset for many organizations, including cloud computing. However, there were some drawbacks that many administrators weren’t aware of when implementing a Hyper-V infrastructure.

For example, there are ramifications of cloning servers without Sysprepping the base image first. Sysprep generate a new SID (the unique Security Identifier each machine has) upon first-time boot up. There are also ramifications with the Virtual host time service, which provides time to the virtual guests, but if the guests are part of an AD infrastructure, or if the guests are DCs, then the host time synchronization service will cause problems with the default AD forest time hierarchy, due to Kerberos’ five minute skew tolerance. Easy enough, it’s recommended to disable time synchronization on the host to prevent this from occurring.

One of the more important ramifications, which we will discuss in the section, involves virtualized snapshots and domain controllers and using the Revert feature to roll back a virtual machine to a previous point in time using a previously saved snapshot. The ramifications can effectively make a DC useless.

In this blog, we’ll talk about:

  • What is a Snapshot?
  • What is the USN?
  • What is a USN Rollback?
  • Windows Server 2012 Snapshot Support
  • Windows Server 2012 Cloning Support

What is a Snapshot?

Hyper-V provides the ability to create a point-in-time copy of a virtual guest. The point-in-time copy is called a snapshot. The snapshot can be used to “revert” the virtual guest back to the point-in-time the snapshot was created.

Snapshots are a convenient means to return a virtual machine to a previous state, such as to return to a state prior to installing an application that is no longer behaving properly.

What is the USN?

The USN, or Update Sequence Number, is the basis of how Active Directory Replication works.

The USN is a value stored with each attribute that changes by either a local change, or a replicated change from a partner domain controller. Each domain controller keeps track of its own changes, and other domain controllers in the infrastructure are aware of all other domain controller USN value.

Active Directory replication relies on Update Sequence Numbers (USNs) on each domain controller. The USN acts as a counter. Each DC’s USN value is unique to a domain controller. The replication system is designed with this restriction in mind.

When an inbound replication partner domain controller sees its partner has a higher USN value for any attribute, a replication pull request is made to replicate the changes to the partner.

Active Directory Replication does not depend on or use time displacement or a time stamp to determine what changes need to be propagated. Time based propagation as some directory services use, are based on a time stamp with the “last writer wins” rule, however this can pose a problem if the clock were to be rolled back.

A time stamp is used in Active Directory, but it’s only used to determine and resolve a conflict when an attribute has been modified at two different DCs simultaneously. In this case, the DC receiving the update will use one of three values to resolve a conflict:

  1. The Version number that is incremented on an attribute by the original writer
  2. The originating time of the original writer
  3. The originating DSA value, which is the GUID of the domain controller (found in ADSI Edit and in the DNS zone).

And because these USN counters are local to each DC, it ensures and is determined that the USN to be reliable by replication partner DCs, because the local DC keeps track of all its own changes.

The USN can never “run backward” (decrease in value). If it does, replication partner DCs will recognize the decreased value, and determine it as an inconsistency, and will remove the DC from its own replica set. This is called a USN Rollback. Although they can be repaired, in many cases, it’s easier and more time efficient to simply force remove the DC with the USN Rollback, and re-promote it back into the domain.

You can use Ldp.exe or ADSI Edit to read the current USN, which is the highestCommittedUsn attribute that can be found on the RootDSE object properties for the domain controller.

Up-to-Dateness, High-Watermark, Propagation Dampening, InvocationID

Replication takes into account specific values and follows a pre-defined algorithm to insure replication consistency among domain controllers to reduce or eliminate divergence, such as the following:

Up-To-Dateness vector

  • This is a value that the destination domain controller maintains for tracking the originating updates that are received from all source domain controllers.
    • This value helps the source DC filter irrelevant attributes (and entire objects if all attributes are filtered) on the basis of the relationships between all sources of originating updates and a single destination.
    • To see the Up-to-datenes vector value, run the repadmin /showvector command.


  • This is a value that the destination domain controller maintains to keep track of the most recent change that it has received from a specific source domain controller for an object in a specific directory partition.
    • This value prevents irrelevant objects from being considered by the source domain controller with respect to a single destination.
    • To see the value of the High-watermark, run repadmin /showreps /verbose and look for each line that starts “USNs:”. The high-watermark USN is the number that is followed by “/OU”.

Propagation Dampening

Fault tolerance is helpful by installing multiple DCs, and provides multiple replication paths between them to reduce latency; however, you might expect the same replication change to be replicated in an endless loop. The Up-to-dateness vector eliminates this possibility along with the InvocationID. The InvocationID of a domain controller and its USN combined provides a unique identifier in the forest associated with every write-transaction performed on each domain controller.

Replication example

To understand the consequences of snapshots prior to Windows Server 2012 requires a brief explanation and basic understanding of how Active Directory replication works.

Scenario: Single Domain, single AD Site, three DCs. DC-A, DC-B, and DC-C, all are replication partners between each other.

Replication Steps

  • DC-A updates a password. The USN is set to 3.
  • DC-B detects a USN change on DC-A
  • DC-B requests the change from DC-A
  • DC-B sends its high-watermark and up-to-dateness vector to DC-A

—–> DC-A looks at the high-watermark and up-to-dateness vector values, and the object that was changed, (the password attribute).
—–> DC-A sees that the originating DSA for the password change is DC-A (itself).
—–> DC-A reads the up-to-dateness vector from DC-B and finds that DC-B is guaranteed to be Up-To-Date from the change from DC-A (itself), but has a USN value of 2.
—–> DC-A sees that the originating USN is 3 on that password attribute.

  • Based on the fact 3 is greater than 2, DC-A sends the changed password to DC-B.


In summary, propagation dampening occurs if DC-B already received the changed password from DC-C, which received it from DC-A, therefore, DC-B will not request the changed password from DC-A.

Additional reading, and summarized from:
Tracking Updates (Active Directory Replication)


Pre-Windows 2012 Virtualized DC recommendations

  • Do not take snapshots or revert back to a snapshot of a domain controller virtual machine.
  • Do not copy the domain controller VHD file.
  • Do not export the virtual machine that is running a domain controller.
  • Do not restore a domain controller or attempt to roll back the contents of an Active Directory database by any other means than a supported backup solution.


Undetected USN Rollback

From: Running Domain Controllers in Hyper-V


Detected USN Rollback

From: Introduction to Active Directory Domain Services (AD DS) Virtualization (Level 100)


Repairing USN Rollbacks

To repair a USN Rollback may be difficult. You can use the replication monitoring and diagnostic tools to determine the extent of the damage. If severe where the USN Rollback is undetected, such as when the VHD file attached to a different virtual host is copied and run on another virtual host, which will make it extremely difficult to determine the cause due to duplicate DC SID numbers, besides the rollback, or if the USN on a restored DC has increased past the last USN that the other domain controller has received. In this case, the USN values of the originating DC are different than what the replication partner believes they should be.

The easiest way to repair a USN rollback is to force remove the domain controller that was reverted, run a metadata cleanup to remove the domain controller’s reference from the AD database, and re-promote it.

Reverting back to a snapshot can cause ramifications with other types of services. For one, you must keep in mind of the secure channel that is used by Active Directory members to communicate to the domain. The secured channel uses a password that gets renewed every seven days. For example, if you revert the machine back prior to the point with a previous password, it may no longer be able to communicate. To repair such a scenario, you can reset the machine account, or disjoin it then rejoin it back to the domain. For servers, such as a Microsoft Exchange server, the implications can be much deeper. Besides the secured channel, users will lose any emails that were received between the current time and snapshot time.


Windows Server 2012 Snapshot Support Prevents USN Rollbacks

Until the introduction of Windows Server 2012, cloning, snapshotting, or copying, are unsupported. The only supported method to repair a DC is to potentially either using Windows Backup, or a third party backup that supports non-Authoritative or Authoritative restores, or simply force demote and rebuild the DC from scratch and promote it back into the domain. Otherwise, as we’ve discussed, snapshots and cloning have serious ramifications that can result in USN rollbacks or lingering objects, just to name a few.

Windows Server 2012 now supports DC cloning and snapshot restore of domain controllers. The requirements to support the new feature are:

  • Hypervisor that supports VM-GenerationID. Window Server 2012 Hyper-V supports VM-GenerationID. If using a third party Hypervisor, check with the vendor if their latest version supports this feature.
  • The source virtual domain controller must be running Windows Server 2012.
  • A Windows Server 2012 PDC Emulator FSMO Role must be running and available for the cloned DC.


How does the VM-GenerationID work?

When you promote a domain controller in a supported Hypervisor, AD DS stores the VM-GenerationID (msDS-GenerationID attribute) in the DC’s computer object in the Ad database. This attribute will now be tracked by a Windows driver in the virtual machine.

If you revert to a snapshot, the driver looks at the current VM-GenerationID value and compares it to the value in the AD database on its computer object. The comparison also occurs each time a DC is rebooted.

If the VM-GenerationID are different:

  • The InvocationID is reset
  • The RID pool is deleted
  • The new value is updated in the AD database, thus preventing any possibility of the USN values to be re-used.
  • A non-authoritative SYSVOL synchronization occurs to safely restore and re-initialize SYSVOL (to prevent a JRNL-WRAP error).
  • Each time a DC is rebooted, the value is compared, and if they are different, this rule and action applies.
  • These actions also safeguards shutdown virtual DCs.

If the VM-GenerationID are the same:

  • The snapshot and transaction is committed.


Windows Server 2012 Cloning

In Windows Server 2012, administrators no longer need to use Sysprep to clone a machine, promote it to a domain controller, then complete any additional tasks such as Windows Updates, or install organization standard applications. After the first domain controller is freshly installed from scratch or using Sysprep in a domain, Administrators can now safely deploy cloned domain controllers by simply copying an existing virtual domain controller.

This feature is domain specific. A domain must have at least one DC installed that can be copied. You still want to properly configure DNS settings, validate each DC’s health, replication status, and run the Active Directory Best Practice Analyzer after each Dc deployment.

This feature provides the following advantageous and benefits:

  • Rapid DC deployment
  • Quick restores
  • Optimize private cloud deployments
  • Rapid DC provisioning to quickly meet increased capacity needs

What if I Don’t Want the VM-Generation ID Mechanism to Kick In?

Perhaps there’s a time when you don’t want this protection, such as if you are trying to clone your environment to a lab. If you follow the rules, the VM-Generation ID will protect the USN and probably not give you what you want, and worse, if the DCs you’re trying to clone are having trouble replicating SYSVOL, you have more problems to deal with.

One way around it to prevent the VM-Generation ID to kick in at the hypervisor level is to shut down the VMs, and simply do a flat file copy to another hypervisor, then create a new VM from using the existing files.That should help the attribute mechanism from kicking in. More info on this and other thoughts:

Cases where VM-GenerationID doesn’t help make Active Directory virtualization-safe -Part 1

Why Windows Server 2012 AD VM-Generation ID functionality is not an alias for Active Directory anti-USN Rollback functionality


Additional Reading:

Tracking Updates (USN & Active Directory Replication)

Running Domain Controllers in Hyper-V

How to detect and recover from a USN rollback in Windows Server 2003, Windows Server 2008, and Windows Server 2008 R2

Steps for deploying a clone virtualized domain controller

Virtual Domain Controller Cloning in Windows Server 2012

By Ace Fekay

MCT, MVP, MCSE 2012/Cloud, MCITP EA, MCTS Windows 2008/R2, Exchange 2007 & 2010, Exchange 2010 Enterprise Administrator, MCSE 2003/2000, MCSA Messaging 2003
  Microsoft Certified Trainer
  Microsoft MVP: Directory Services
  Active Directory, Exchange and Windows Infrastructure Engineer

Comments are welcomed.

How to Recover a Journal Wrap Error (JRNL_WRAP_ERROR) and a Corrupted FRS SYSVOL from a Good DC – What option do I use, D4 or D2? What’s the Difference between D4 and D2?

Original: 11/21/2013
Updated 8/30/2014


Ace here again. I’ working on updating all of my blogs. If you see any inconsistencies, please let email me and let me know.


Are you seeing Event ID 13508, 13568, and anything else related to SYSVOL, JRNL_WRAPS, or NTFRS?

Note – I will not address Event ID 2042 or 1864. That’s an issue with replication not working beyond the AD tombstone. If you are seeing them, you’re best bet is to forcedemote the machine, run a metadata cleanup, and re-promote it, and make sure you configure your firewall and/or AV to allow replication traffic or stop using the ISP’s or router as a DNS address, or disable IP routing and WINS Proxy, to prevent this in the future. And while you’re at it bump up your AD tombstone to 180 days,

As for the NTFRS, after talking to numerous folks whether directly assisting a customer, or through the TechNet forums, there seems to be some confusion associated with how to handle Journal Wrap errors, what caused them, and what are the differences between the D2 and D4 options. I’ll try to quell this confusion in this blog, as well as provide an easy step-step and providing an explanation for the steps, to get out of this error. Note: The steps are from Microsoft KB290762. I just thought to further break it down so a layman will understand them.

Reference KB: Using the BurFlags registry key to reinitialize File Replication Service Replica Sets

For Windows 2008/2008 R2/2012/2012 R2 with DFSR

Follow this KB to fix it:

How to force an authoritative and non-authoritative synchronization for DFSR-replicated SYSVOL (like “D4/D2” for FRS)

Backing Up and Restoring an FRS-Replicated SYSVOL Folder 

What Caused the Journal Wrap?

First you have to ask yourself, what caused this error on my DC? What did I do to get here? In a nutshell, JRNL_WRAPS are caused by SYSVOL corruption.

The usual culprit can be a number of things:

  • Abrupt shutdown/restart. I don’t usually see this unless there are power issues in the building with not power protection or UPS battery system.
  • Disk errors – corrupted sectors. This is a common issue with a DC on older hardware.
  • AV not configured to exclude SYSVOL, NTDS and the AD processes. This is the typical culprit I’ve seen in many cases.

Ok, So what do I have to do to fix this?

To get yourself out of this quandary, it’s rather simple. Yea, you might say yea, right, this is not so simple, but it really isn’t that hard. It just requires a little understanding of what you have to do, which is all it’s doing is simply copying a good SYSVOL folder and subfolders from a good DC to the bad DC (the one with the errors.

Basically, you first choose which DC is the good DC to be your “source” DC for the SYSVOL folder. Then you you stop the NTFRS service on all DCs. Yes, NTFRS must to be stopped on all DCs to perform this. Then set the registry key on the good DC and the bad DC. That’s it. The process will take care of itself and reset the keys back to default after it’s done.

  • If you only have one DC, such as an SBS server, and SYSVOL  appears ok, or restore just the SYSVOL from a backup. Then just follow the “Specific” steps I’ve outlined below.
  • If more than one DC, but not that many where you can’t shutdown the NTFRS on all of them, such as if you have 40 DCs, pick and choose the best one and set Burflags to D2 on the bad and D4 on the good.
  • If there are numerous DCs, such as a large infrastructure, simply run dcpromo /forcedemote the DC with the error, run a metadata cleanup, then re-promote to a DC back into the domain. If you unplug the DC and run a metadata cleanup, then you will have to rebuild the DC from scratch. The forcedemote switch removes the AD binaries off the machine allowing you to re-promote it.


To summarize:

You have two choices as to a restore from a good DC using FRS:

  1. D2 is set on the bad DC: Non-Authoritative restore: Use the D2 option on the DC with the empty SYSVOL folder, or the SYSVOL folder with the incorrect data. This way it will get a copy of the current SYSVOL and other folders from the good DC that you set the BurFlags D4 option on.
  2. D4 is set on the good DC: Authoritative restore: Use the BurFlags D4 option on the DC that has a copy of the current policies and scripts folder (a good, not corrupted folder).


The BurFlags option – D4 or D2? What do I use?

The steps refer to changing a registry setting called the BurFlags value. If the BurFlags key does not exist, simply create it. It’s a DWORD key.

More importantly, it references change the BurFlags to one of two options: D4 or D2. Therefore, before going further, I would like to squelch the confusion on what the D2 and D4 settings mean:

D2/D4 – Which is which?

  • D2, also known as a Nonauthoritative mode restore – this gets set on the DC with the bad or corrupted SYSVOL
  • D4, also known as an Authoritative mode restore – use this on the DC with the good copy of SYSVOL.
  • You must shut the NTFRS service down on ALL DCs while you’re doing this until instructed to start it.
  • You’ll probably want to copy the current SYSVOL structure on the good DC to another folder as a backup prior to doing this.

The D2 option on the bad DC will do two things:

  1. Copies the current stuff in the SYSVOL folder and puts it in a folder called “Pre-existing.” That folder is exactly what it says it is, it is your current data. This way if you have to revert back to it, you can use the data in this folder.
  2. Then it replicates (copies) good data from the GOOD DC (D4) to the bad guy (D2).

Once again, simply put:

  • The BurFlags D4 setting is “the Source DC” that you want to copy its good SYSVOL folder from, to the bad DC.
  • The bad DC BurFlags is set to D2, which tells it to pull from the source DC, the one you set D4 on.


Here are the steps summarized:

  1. For an Authoritative Restore you must stop the NTFRS services on all of your DCs
  2. In the registry location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process
    1. Set the BurFlags setting to HEX “D4” on a known DC that has a good SYSVOL (or at this time restore SYSVOL data from backup then set the Burflag to D4)
    2. Then start NTFRS on this  server.
    3. You may want to rename the old folders with .old extensions prior to restoring good data.
  3. Clean up the folders on all the remaining servers (Policies, Scripts, etc) – renamed them with .old extensions.
  4. Set the BurFlags to D2 on all remaining servers and then start NTFRS.
  5. Wait for FRS to replicate.
  6. Clean up the .old stuff if things look good.
  7. If the “D4” won’t solve the problem try the “D2” value.


So circling back, to fix this and make it work, just copy the contents of SYSVOL to another location, then follow the KB, which simply states you must stop the NTFR service on ALL DCs. Then pick a good one to be the “Source DC.”

Of course, as I’ve stated above, if you have a large number of DCs, the best bet is to forcedemote the bad DC, run a metadata cleanup to remove its reference from AD, then re-promote it.

If you have a small number of DCs, and if you have a good DC and a bad DC, on the good DC, you would set the BurFlags to D4, and on the BAD DC you would set the Burflags to D2.

Example run:

In the example below, if you set BurFlags to D4 on a single domain controller and set BurFlags to D2 on all other domain controllers in that domain, you can rebuild the SYSVOL from the D4 DC (the source DC).

I’ve also heard of admins manually copying the SYSVOL folder, then set the BurFlags options as mentioned, which works too. But no, I haven’t tested it. That would be for a lab on another day. 🙂

Authoritative Restore Example

Use the BurFlags D4 option on the DC that has a copy of the current policies and scripts folder (a good, not corrupted folder).

  1. Stop the FRS service on all DCs. To do this to all DCs from one DC, you can download PSEXEC and run “psexec \\otherDC net stop ntfrs” one at a time for each DC.
  2. On a good DC that you want to be the source, run regedit and go to the following key:
    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup
    In the right pane, double-click “BurFlags.” (or Rt-click, Edit DWORD)
       Type D4 and then click OK.
  3. On the bad DC, run regedit and go to the following key:   HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup
       In the right pane, double-click “BurFlags.” (or Rt-click, Edit DWORD)
       Type D2 and then click OK.
  4. Quit Registry Editor, and then switch to the Command Prompt (which you still have opened).
  5. On the good DC, start the FRS service, or in a command prompt, type in “net start ntfrs” and hit <enter>
  6. On the bad DC, start the FRS service, or in a command prompt, type in “net start ntfrs” and hit <enter>
  7. On the bad DC, check the Sysvol folder to see if it started populating.
  8. Check for EventID 13565 which shows the process started
  9. Check for EventID 13516, which shows it’s complete
  10. Start FRS on the other DCs.

The following occurs after running the steps above after you start the FRS service (NTFRS):

  • The value for BurFlags registry key returns to 0.
  • Files in the reinitialized FRS folders are moved to a <var>Pre-existing</var> folder.
  • An event 13565 is logged to signal that a nonauthoritative restore is started.
  • The FRS database is rebuilt.
  • The member replicates (copies) the SYSVOL folder from the GOOD DC.
  • The reinitialized computer runs a full replication of the affected replica sets when the relevant replication schedule begins.
  • When the process is complete, an event 13516 is logged to signal that FRS is operational. If the event is not logged, there is a problem with the FRS configuration.
    Note: The placement of files in the <var>Pre-existing</var> folder on reinitialized members is a safeguard in FRS designed to prevent accidental data loss. You can copy this stuff back if it didn’t work, but I have not yet seen when this has not worked!


I hope this helps cleaning up your FRS and SYSVOL replication issues.

Ace Fekay
MVP, MCT, MCSE 2012, MCITP EA & MCTS Windows 2008/R2, Exchange 2013, 2010 EA & 2007, MCSE & MCSA 2003/2000, MCSA Messaging 2003
Microsoft Certified Trainer
Microsoft MVP – Directory Services
Complete List of Technical Blogs:

This blog is provided AS-IS with no warranties or guarantees and confers no rights.