Windows Server Clustering & PCNews

Several terms that come up almost daily with my customers. Defined here for you:

Supported – We have a wide range of support options. When we (Microsoft) state we support something, it really means we have tested that scenario and that is works as we programmed it to.

Best Effort – As defined here http://support.microsoft.com/kb/926172. PSS support for hardware-related/network-related/third party software issues is limited to a commercially reasonable effort. If PSS determines that the issue is related to the hardware/network/third party software, you must obtain support directly from the vendor of the computer or from the hardware manufacturer.

Unsupported – We have NOT tested this scenario or product usage and because of that we will not be able to help a customer in this state. The product may or may not work as expected; we simply don’t know and haven’t test it.

An unsupported version of Windows will no longer receive software updates from Windows Update. These include security updates that can help protect your PC from harmful viruses, spyware, and other malicious software, which can steal your personal information. Windows Update also installs the latest software updates to improve the reliability of Windows—new drivers for your hardware and more. (from http://windows.microsoft.com/en-us/windows/help/what-does-end-of-support-mean)

Recommended – We support and recommend this configuration. We have experience supporting other customers doing exactly what you are doing or have purposed to do. This is may or may not be the best way of using or configuring the product, but we are ok with your approach – in other words it’s not a bad idea.

Best Practice – We support, recommend, and have experience that over time this is the best way to configure or use the product.  This does not mean this is the only way to use or configure the product, we will potentially support other ways of product use/configuration. We either use this internally and/or have other large customers using or configuring the product this way.

Worst Practice – We never in a million years thought a customer would do what are or have purposed to do with the product. We haven’t tested or designed the product to be used in this way. In the grand scheme of things this is a very, very bad plan and should be corrected to a recommended or best practice ASAP. We won’t support this configuration or usage. Please reconsider your choice.

Now go out there and do something that is a best practice today 🙂

1)      Copy robocopy.exe from the Resource Kit to something like c:\backup\tools. This is used in step 5 below.

2)      Create a backup list using the GUI version. Save a system state only as night.bks, save all the drives and system state as night-all.bks. Make sure you use the backup directory path listed below in the script or modify to a SAN location.

3)      Use the follow for the required .bat file

C:\backup\tools\nightlybackup.bat:

            *** BEGIN BAT FILE ***

@ECHO OFF

REM

REM  Build Date String yyyymmddhhmm

REM

for /F “usebackq tokens=2” %%i IN (`DATE /T`) DO Set TIMESTR=%%i%

for /F “usebackq tokens=1,2,3 delims=/” %%i IN (‘%TIMESTR%’) DO SET MONTH=%%i%

for /F “usebackq tokens=1,2,3 delims=/” %%i IN (‘%TIMESTR%’) DO SET DAY=%%j%

for /F “usebackq tokens=1,2,3 delims=/” %%i IN (‘%TIMESTR%’) DO SET YEAR=%%k%

for /F “usebackq” %%i IN (`Time /T`) DO Set TIMESTR=%%i%

for /F “usebackq tokens=1,2* delims=:” %%i IN (‘%TIMESTR%’) DO SET HOUR=%%i%

for /F “usebackq tokens=1,2* delims=:” %%i IN (‘%TIMESTR%’) DO SET MINUTE=%%j%

SET DIRSTR=%YEAR%%MONTH%%DAY%%HOUR%%MINUTE%

SET TIMESTR=

SET MONTH=

SET DAY=

SET YEAR=

SET HOUR=

SET MINUTE=

@ECHO Directory String %DIRSTR%

 

IF NOT EXIST “c:\backup\tools\nightlybackup.bat” GOTO SKIP

 

C:\WINDOWS\system32\ntbackup.exe backup “@\\DOMAINCONTROLLERNAME\c$\Backup_DCNAME\night-all.bks” /n “DCNAME.bkf” /d “Nightly” /v:yes /r:no /rs:no /hc:off /m normal /j “Nightly” /l:s /f “\\DOMAINCONTROLLERNAME\C$\Backup_DCNAME\%DIRSTR%DCNAME.bkf“

 

EXIT

 

:SKIP

 

C:\WINDOWS\system32\ntbackup.exe backup “@\\DOMAINCONTROLLERNAME \c$\Backup_DCNAME \night.bks” /n “DCNAME.bkf” /d “SS_Nightly” /v:yes /r:no /rs:no /hc:off /m normal /j “SS_Nightly” /l:s /f “\\DOMAINCONTROLLERNAME\C$\Backup_DCNAME\%DIRSTR%DCNAME.bkf“

 

EXIT

            *** END BAT FILE ***

 

Comments and legend for the above batch file.

 

First I turn of echo

Next section builds the current date, we will use this to sort the backup and delete ones older than 7 days.

If not exist statement is useful if you store anything on a SAN drive. If they SAN is down normally the backup will fail, so I check before and then just run night to just backup the system state.

First backup runs a backup of the DOMAINCONTROLLERNAME (replace with a real name), creates a new file called DCNAME.bkf (replace with server name), next I label the backup Nightly, verify is then turned on, restricted access is NOT turned on – you may want it though (so /r:yes would turn it on), hardware compression is turned off, the backup type is set to normal, next the job name is set for backup reporting, next logging is set to summary, and finally the file name is created with the date and time stamp at the front (again for sorting and deleting older files later).

 

4)       To schedule a task to automatically backup a machine. Go to Start-> Control Panel -> Scheduled Tasks -> Add Scheduled Task.  During the configuration the following options are selected:

 

Scheduled Task Wizard	Next
Scheduled Task Wizard Application	Browse C:\backup\tools\nightlybackup.bat Open
Scheduled Task Wizard Type a name for this task Perform this task	Nightly Daily Next
Scheduled Task Wizard Start Time Perform this task Start date	3:15 AM Every Day (today) Next
Scheduled Task Wizard Enter the user name Enter the password Confirm password	Next
Scheduled Task Wizard Open advanced…	Unchecked Finish

 

5)      Finally I use robocopy to clean up, so the backup won’t fill up the disk.

c:\backup\tools\robocopy c:\backup\ c:\backup\t *.bkf /MOV /MINAGE:10 rd /q /s c:\backup\t

I use Robocopy to look at the c:\backup and create a c:\backup\t folder (for temporary). I look for anything that ends with .bkf which is then Moved to the temporary (c:\backup\t) folder, if it is at least 10 days old. This process is done quietly and uses a sub directory of c:\backup\t.

 

Enjoy, let me know if you have any questions or issues.

The Windows Server Failover Clustering Product Group is proud to announce the release of the first ever Windows Server 2008 Microsoft Official Curriculum course

Course 6423A: Implementing and Managing Windows Server 2008 Clustering

 

The course will be available in the channel May 15^th, 2008.

 

Special thanks go to the following to make this happen:

 

Elden Christensen

Ahmed Bisht

Chuck Timon

Mike Briggs

 

I have seen it, excellent work gang, this is truly an exciting day for Failover Clustering!!!

 

 

As you might we aware of I recently joined Microsoft as a Senior Premier Field Engineer. While I don’t work out of the Redmond campus, I have been there numerous times. Here is a great humorous look at life here at Microsoft:

 

http://on10.net/blogs/tina/Life-At-Microsoft/

 

Enjoy 🙂

Ok, it is not new, but now it gets new posts weekly.

 http://blogs.msdn.com/clustering/

 They even mentioned this blog today http://blogs.msdn.com/clustering/archive/2008/01/10/7063069.aspx

Look to it for tons of great information.

The 2008 articles are just starting to hit the web. This one deals with migration, not upgrading. You can’t upgrade to 2008 Failover Clustering. I am actually glad you can’t upgrade, I really don’t like that word.

 Here is the article: http://technet2.microsoft.com/windowsserver2008/en/library/6820ae3f-1ecc-43fd-8a76-fe8c2125cfe61033.mspx?mfr=true

Are you ready for another OS version? February 27^th, 2008 is the public launch date. Microsoft has a Vista sidebar gadget to help you remember the date – http://www.wedsg.com/winclient/vistasidebargadget/download.htm They seem to love this date, the end of February as they have done it before. Could it be they know by wives birthday is Feb 28? 

Free E-Learning for 2008 – https://www.microsoftelearning.com/eLearning/offerDetail.aspx?offerPriceId=127516 (requires Microsoft Passport login to get the free stuff). 

Links to everything: http://technet.microsoft.com/en-us/windowsserver/2008/default.aspx 

Nice overview: http://www.microsoft.com/windowsserver2008/evaluation/overview.mspx 

2008 Learning Portal: http://www.microsoft.com/learning/windowsserver2008/default.mspx 

Clustering: http://technet2.microsoft.com/windowsserver2008/en/library/13c0a922-6097-4f34-ac64-18820094128b1033.mspx?mfr=true 

File Server Clustering: http://technet2.microsoft.com/windowsserver2008/en/library/40bdcbc9-ce96-4477-8df3-7a20d4bc42a51033.mspx?mfr=true 

Print Server Clustering: http://technet2.microsoft.com/windowsserver2008/en/library/40bdcbc9-ce96-4477-8df3-7a20d4bc42a51033.mspx?mfr=true 

NLB: http://technet2.microsoft.com/windowsserver2008/en/library/40bdcbc9-ce96-4477-8df3-7a20d4bc42a51033.mspx?mfr=true 

I love new OS releases and 2008 looks to be an excellent one!

 

My good buddy Scott Schnoll attempts to clear up some confusion on when and where to use Cluster.exe/CluAdmin to move a CMS. Find his post here:

 http://msexchangeteam.com/archive/2007/10/22/447317.aspx

Here is the comment I left: 

The samples you sited are for creating or managing a cluster, none of them are for moving a CMS.

Using CluAdmin/Cluster.exe vs. EMC/EMS with SP1 require different security levels by default and may not be the same individual. Now we will have to give Cluster Administrator Exchagne rights to avoid “Bad things Happening”…

Thankfully SQL and other Microsoft produts don’t have an issue with CluAdmin/Cluster.exe. Nor do they have any other way to manage them, K.I.S.S. in action.

Thanks for the post Scott!

This past weekend I ran into a few issues with Exchange Server 2007 and wanted to share, so anyone with them won’t have to call Microsoft PSS and go through the fun (ok, not really fun…) that I went through.

Partition in Time with CCR

You have a partition in time, but what does that mean. You lost a node or the witness, and while that
was happening the remaining node/witness thought a change was made. When the
down node/witness came back it detected that a change has occurred and
killed the entire cluster. This is by design. Now, how do you fix it?

http://support.microsoft.com/kb/258078 ForceQuorum section:

Function: When you use a Majority Node Set (MNS) quorum model on a Windows
Server 2003 cluster, in some cases a cluster must be allowed to continue to
run even if it does not have “quorum” (majority). Consider the case of a
geographically dispersed cluster with four nodes at the “primary” site and
three nodes at the “secondary” site. While there are no failures, the
cluster is a seven-node cluster where resources can be hosted on any node,
on any site. If there is a communications failure between the sites or if
the secondary site is taken offline (or fails), the primary site can
continue because it will still have quorum. All resources will be re-hosted
and brought online at the primary site.

In the event of a catastrophic failure of the primary site, however, the
secondary site will lose quorum, and, therefore, all resources will be
terminated at that site. One of the primary purposes for having a multi-site
cluster is to survive a disaster at the primary site; however, the cluster
software itself cannot make a determination about the state of the primary
site. The cluster software cannot differentiate between a communications
failure between the sites and a disaster at the primary site. That must be
done by manual intervention. In other words, the secondary site can be
forced to continue even though the Cluster service believes it does not have
quorum. This is known as forcing quorum.

Because this mechanism is effectively breaking the semantics associated with
the quorum replica set, it must only be done under controlled conditions. In
the example above, if the secondary site and primary site lose communication
and an administrator forces quorum at the secondary site, resources will be
brought online at BOTH sites, thus allowing the potential for inconsistent
data or data corruption in the cluster.

Requirements: Forcing quorum is a manual process that requires that you stop
the Cluster service on ALL the remaining nodes. The Cluster service must be
told which nodes should be considered as having quorum.

Usage scenarios: Special care must be taken if and when the primary site
comes back because the nodes are configured as part of the cluster. While a
cluster is running in the force quorum state, it is fully functional. For
example, nodes can be added or removed from the cluster; new resources,
groups, and so forth can be defined.

Note The Cluster service on all nodes NOT in the force quorum node list must
remain stopped until the force quorum information is removed. Failure to do
so can lead to data inconsistencies OR data corruption.

Operation: Set up the Cluster service startup parameters on ALL remaining
nodes in the cluster. This is done by starting up the Services control
panel, selecting the Cluster service, and then entering the following in the
Start parameters option:
net start clussvc /forcequorum node_list
For example, if the secondary site contains Node5, Node6, and Node7, and you
wanted to start the Cluster service and have those be the only nodes in the
cluster, use the following command:
net start clussvc /forcequorum /forcequorum node5,node6,node7
Note There should be no spaces in the key (except where there are spaces in
the node names themselves).

The only problem I could not get the above commands to work on a 64-bit Windows Server 2003 R2, Enterprise Edition SP2 machine. I most got invalid syntax. Here is what PSS told me to do:

1.    We shutdown one of the nodes, a true power off. We will call this the passive node.
2.    We added the following value to this registry key on the surviving node (active node):

HKLM/System/CurrentControlSet/Services/Clussvc/Parameters

Value: ForceQuorum
Type: REG_SZ
Data: nodenamea

3.    Replace nomenamea with the machines name, such as exch2007nodea – where this is the node that is currently running.
4.    We attempted to start the cluster service on the active- surviving node and it started.
5.    We then stopped the cluster service on the active – surviving node and added nodenameb to the ForceQuorum data value on the surviving node.
6.    We restarted the powered off (passive) machine.
7.    We then started the cluster service on the active node and it started. The registry with the ForceQuorum containing both node names.
8.    We attempted to start the cluster service on passive (with no parameters or registry changes) and it started.
9.    We verified that the Cluster group resources were online.
10.   Undo the registry changes by deleting the ForceQuorum key from the Active node.

Exchange Server 2007 System Attendant fails to come online within a CCR/SCC cluster

After the cluster was up and running, the Exchange SA was not. Looking in the Application event log and we were getting the following errors with regards to the Exchange SA failing to start:

Event ID 1011, 1030, 1003, and 1019 errors.

We found that a bug exists where the Exchange SA times out after 40 seconds when the default of 180 seconds is used for the resource.

We changed the value to 179 and the Exchange SA resource came online. This is scheduled to be fixed in SP1. This bug was confirmed for SCC & CCR Exchange Server 2007 Clusters. 

Update from PSS – find a link to the first issue here http://technet2.microsoft.com/WindowsServer/f/?en/library/e70333db-5048-4a56-b5a9-8353756de10b1033.mspx, we are still waiting on the KB to be updated though.

Sometimes I think that the movie Conspiracy Theory should have been about the software industry today. What has become of it lately? Here is what I believe:

·         I believe the Anti-Virus companies write all the viruses.

·         I believe most software is way over priced.

·         I believe we now alpha test software for vendors

·         I believe we beta test when service pack 1 comes out.

·         I believe 1.0 is not the standard to avoid, RTM (release to manufacturing/gold code) is.

·         I believe we get the final, ready for the world product when service pack 2 comes out.

·         I believe most software has too many features for 98% of the users.

·         I believe all the added features cause 100% of the problems with software today.

·         I believe it is better to update software then to design it properly to begin with.

·         I believe we, the paying consumer, don’t complain enough so things are only going to continue to get worse.

·         I believe you pay 10 times the cost of software in support costs and lost productivity when it does not function properly out of the box.

·         I believe the world has become too computer savvy because of buggy software.

·         I believe a computer should be just another asset at the office place, taken for granted like a stapler or pencil.

·         I believe that a computer isn’t taken for granted because broken things always get attention and notice.

·         I believe release dates are based upon dates on PowerPoint slides, not when the product is anywhere near being ready or bug free.