Windows Server 2008 Server Core R2 Reboot Avoidance

I’ve been a fan of Server Core since I heard about it and you can see by the 20+ posts I have on the product.  Server Core has been out since Windows 2008 which is just under 40 months.  Server Core R2 has been out for about 20 months.  One of the selling points that Microsoft made on server core was that it would have a reduced attack surface and thus you would have fewer reboots due to patches.  While preparing for a talk on Server Core I wanted to investigate this a bit more.  I reached out for some help to Andrew Mason (if you don’t know that name you aren’t a Server Core hard core freak like me).  Andrew runs the official Server Core blog over on TechNet.  Andrew sent me some wonderful information on the subject that I’d like to share with you.

Take a look at these numbers.  I’ll explain in more detail below.

Core

We are comparing both Server Core 2008 and R2.  The Reduction column is % reduced based off the hotfixes Microsoft released during their existing lifespan.  Critical Only is just that, the reduction of patches Microsoft rated Critical for both versions of Server Core.

Now lets look at the rows starting with All applicable patches.
  • All roles are all available roles for those versions of Server Core.
  • Months without a reboot is really cool.  It shows how many months went by with no reboots required on Server Core.  Although it is not consecutive months it is still pretty impressive that Server Core R2 has not needed a reboot during half its existence!
  • Next we see the reduction of patches with the basic OS installed an none of the major features and roles installed.

Now we see the area called Necessary patches only.  What does that mean?  That is referencing the patches that are really needed for Server Core.  There are some vulnerabilities that show Server Core as vulnerable but its not exploitable.  That is what is called out on the bottom of the graphic.  Microsoft does this because it has changed the file and would probably prefer you to update the file eventually too.  IMHO I’d patch these but would bundle them with the necessary patches.

I remember reading an article from David Cross on TechNet stating the following “In some cases, customers can see up to a 60% reduction in patch requirements and the number of reboots on a monthly basis”  These are the numbers that back up statements such as that.

Those are some pretty impressive numbers. Great job to the whole Server Core team.  I really hope Microsoft continues with this product and from recent announcements on the next version of SQL it looks like they are sticking with it.

*Numbers updated through the May 2011 patch Tuesday

A Couple Quick Active Directory One-Liners

Here is a few one-liner commands to help get info on your Active Directory environment.  I don’t think there is any mind blowing commands here but they’ve helped me out.  There are literally hundreds of these around the web as well as PowerShell ones but these are the ones that I’ve been using lately. 

How to view the Domains you trust and see what those Domain SIDs are:

nltest /domain_trusts /v

A quick listing of your AD Sites:

dsquery * "CN=Sites,CN=Configuration,DC=forestRootDomain" -attr cn description location -filter (objectClass=site)

A quick listing of your AD sites and their Site Links and Costs (sure would be nice if you could spit this out to Visio or something):

dsquery * "CN=Sites,CN=Configuration,DC=forestRootDomain" -attr cn costdescription replInterval siteList -filter (objectClass=siteLink)

Compare time against your forest root PDCe:

w32tm /monitor /computers:ForestRootPDC

Find out which DC for a site is the ISTG:

dsquery * "CN=NTDS Site Settings,CN=siteName,CN=Sites,CN=Configuration,DC=forestRootDomain" -attr interSiteTopologyGenerator

Everything you wanted to know about Active Directory Replication but were afraid to ask

I was thinking about writing a post about Active Directory replication but thankfully soon realized that by doing so I could be severely depriving my kids and wife of a happy life.  Its not that Active Directory replication is bad or harmful, its just that there is so much about it.  I don’t care who you are you probably don’t know it all…I certainly don’t and have never claimed too.

While I was doing my research for this post I found what I”d like to call the bible to Active Directory replication.  I’m also thankful this was one of the first resources I picked up on and didn’t have to much time invested.  Without further ado – How the Active Directory Replication Model Works.  I think if you printed this out it would be about 100 pages or so (not confirmed but it is long). 

This article goes over every little detail needed to fully understand the Active Directory replication model.  I’d love to know the person/team that wrote this and give them my gratitude.  I wish stuff like existed for new products.  I still remember trying to learn Active Directory when it was in beta back in 1999 and not fully understanding USNs, Up-to-Dateness Vectors and Watermarks.

If you have any good resources on Active Directory replication please feel free to share so others can learn as well.

Find and Disable Stale User Accounts

Stale user accounts can be a big problem…even more so when they are not disabled.  I’m a firm believer that if you have an account that is not being used it should be disabled.  However depending on the size of your Active Directory that can be a daunting challenge.  Below you will find a snippet of code that will identify where user accounts are not being used for 10 weeks and then it has the ability to disable them. 

dsquery user -inactive 10 -limit 0

The 10 value is for the number of weeks an account has been inactive.  If you think you are going to have a lot of these then you may want to change your limit from 0 to something like 50 or so.

Now if you would like to disable them as well you simply add on another portion of code.  For safety reasons I prefer to run the code above first to see who is inactive and then once I’ve validated those accounts can be inactive I run the following code to disable them.

dsquery user -inactive 10 -limit 0 | dsmod user -disabled yes

Obviously the account needs to have the appropriate permissions for dsmod to work so watch out for that.  Good luck and happy hunting!

How to Delegate the Right to Delegate Kerberos Constrained Delegation

Wow, that is a lot of delegating…seriously how many times can you say it in one sentence.  Today’s post is one that threw me for a loop.  As a domain admin I have the right to configure constrained Kerberos delegation.  There may come a time when you want to delegate that out to a user or group. 

My first thought was to assign the user/group Full Control on the OU that included the accounts.  At this point I would run the following command

setspn -a http/workstation01 adminprepbrian

Surely Full Control would grant me the permission to do this…Failed!!!  Insufficient access rights.  It is not a “permission” that is needed, it is a “User Right”.  So where do you go to assign rights to work with constrained delegation and what User Right is it?  Well, you won’t find it in the Local Security Policy.

The User Right that you need to grant is SeEnableDelegationPrivilege. Now where and how do I grant this User Right.  Well it turns out you still should delegate Full Control to the user/group that you want to grant this User Right too.  Then on a DC you must run the following command:

ntrights -u adminprepbrian +r SeEnableDelegationPrivilege

Just make sure to modify that domain/user to match your environment.  Now when I run the Setspn command it works because that account has the correct User Right.  You may have to wait for replication to occur if you are in a distributed environment.

What are Service Principle Names (SPNs)?

SPNs seem to get more and more use these days so I thought it be nice to give an explanation of what SPNs are.

SPNs are used for mapping a service to a user account. You will find SPNs used predominantly with Delegation and Impersonation and a lot of times this is between a web server and another server hosting a service that requires Kerberos authentication.  The key here is that Kerberos authentication is required and thus this is primarily used within an organization or a trusted company.  An example of this would be when an end user logs on to a web server which then logs on to a SQL server.  The web server is trying to authenticate against the SQL server using the web users credentials but it doesn’t have the right to do that type of delegation.  If that were the case I don’t think online banking would be…well online.  :,,)  Now this is only the case when the web and SQL instances are on separate servers.  If they were on the same server you would not need to worry about SPNs.

Kerberos is the key here.  Kerberos authentication happens all the time and is very common.  The special part of Kerberos authentication is that it requires a ticket that ensures each party is who they say they are.  This ensures that a hacker can’t impersonate another user.  The only type of delegation that Windows allows is a Kerberos connection.  In short the user knows how to contact and authenticate with the web server but has no idea who the SQL server is but needs data from it and needs to authenticate…thus delegation and impersonation needs to occur.

An SPN is a name that Kerberos clients use to identify a service for computer that is also using Kerberos.  In fact you can have multiple instances of a service running on a system and each could have its own SPN. SPNs have a specific format that they use which looks similar to this – <service class>/<host>:<port>/<service name>  The only parts that are required are the serviceclass and host.  For example, HTTP/www.adminprep.com would be an SPN registration for any page on that webpage.  You would use the port option if you wanted to specify a port with the service, like this – MSSQLSvc/sqlservername.adminprep.com:3411.  More info on the formatting of SPNs can be found here.

SPN names can use short NetBIOS names or long FQDN names.  I recommend always using FQDNs as you can have potential name conflicts in a multi-domain forest with short names.

For a more detailed looked into SPNs i’ve provided a few links below along with links to common issues.  However the first place you should go is to this TechNet article.

Service Principle Name (SPN) Resources and Issues

Lock Your Workstation

I’m sure you are like me when it comes to locking your desktop.  You ALWAYS do it.  Most if not all corporations today have a group policy in place that at least sets the Screen Saver on after a certain amount of time and requires a password for security reasons (User Configuration – Administrative Templates – Control Panel – Personalization – Password protect the screen saver).

You know as well as I do that there is always that one person that seems to always forget to lock their workstation.  Sure the group policy will kick in…eventually.  During that time the system is unlocked and the data vulnerable.

Since i’m such a huge fan of shortcuts I have two for the price of one today.  I will show you two methods to lock your workstation…even for those very forgetful people.

Method 1 (and what I think is the easiest)

By pressing the Windows key and L on the keyboard you effectively lock the system.  I use this one ALL the time.  It is the quickest method that I know.  However some people are not so keyboard shortcut friendly.

Method 2

For the people that prefer to use their mouse here are several steps to create a desktop shortcut.  This method is very similar to the post I had on creating a shortcut for the Network Properties in Server 2008.

1. From where ever you want the shortcut create, Right click and select New –> Shortcut  (I recommend the Desktop)

 

2. Put the following path into location rundll32.exe user32.dll,LockWorkStation



3. Click Next and type whatever you would like the name of the Shortcut Icon to appear as and click Finish.



4. Time to change the way the Icon looks – Right Click on the newly created Shortcut and select Properties

 

5. Click the Change Icon… button and change the path to %SystemRoot%system32SHELL32.dll and now pick whichever Icon you prefer.

 

6. We finally have an icon available to lock the workstation on the Desktop.

 

I personally love when people at work leave their workstations unlocked.  Like a lot of you i’m sure you like to teach that person a lesson.  Perhaps mess with the background…a nice screensaver message on how much they look up to me!

Server Health Checks

I’d like to share some of the things I look at while do a health check on a server.  Its funny how few resources there are out there on the Internet.  I believe people keep this kind of stuff to them self because they are scared they are going to miss something and they will never live it down.  My response to that is, So What!  Heck, I don’t claim to know it all but why not share what I do know and maybe others can share via the Comments!!!

When I’m troubleshooting I like to compartmentalize what I”m looking for.  With that my health checks are set up the same way.  I also believe health checks are quick snapshots of the health of a server.  Sure there are tools that you can use to analyze systems further but in this case we are doing a quick health check.  Not all of these need to be done but some should, you get to decide.

CPU

Occasional high CPU spikes are ok as long as you are aware of the process causing this. A server should maintain 80% CPU utilization for an extended period of time.  If it does it may be time to upgrade.  Its a good idea to keep Task Manager open during the duration of your troubleshooting to see trends.

Check CPU Usage

  1. Open Task Manager

  2. Check the Processes tab, ensure there are no processes consuming excessive CPU

  3. Check the Performance tab, ensure there are no single CPU’s that have excessive CPU usage

Check CPU HW

  1. Open Device Manager (right click computer –> Manage)

  2. Ensure that no CPU’s have red X or yellow ! underneath the Processors

Processes

This is one area that you may not want to do for quick health checks but is something you should be familiar with.  Task Manager only gives you basic info on processes and you will find that you may need to dig a bit deeper.  For that I recommend Process Monitor from the great SysInternal tools.  Process Explorer can also be used.  In fact download and play with all these tools…they will save your bacon, I guarantee it.

In-Depth Check
SysInternals:

Copy Process Monitor locally, then launch it.

  1. Analyze each process and watch what operations open the reg keys, file etc.

Copy Process Explorer locally, then launch it.

  1. Analyze each process based upon the number of threads, handles, loaded DLL’s,etc.

Two great webcasts can be viewed here to see these types of tools in action.

Memory

General rule of thumb is to make sure the general memory utilization does not exceed 80%within a given period of time.

Check Memory Availability

    1. Open Task Manager
    2. Select the Performance tab

    3. Look at the Physical memory box,and multiply the total memory by .2

    4. If the total available memory is less than this number then the box is currently utilizing more than 80 percent of the memory.

Current utilization by process

  1. Select the Process tab

  2. Check the ‘show processes from all users’ box in the bottom left corner

  3. Click the column header ‘Mem Usage’ to sort the processes by memory utilization, highest to lowest. This will help you determine what processes are currently utilizing the memory on the box and can help you narrow your search for memory intensive processes.

Network

Check NIC HW

  1. Verify both ends of the network cable are securely seated in the port

  2. On the back of the server verify you have a green blinking link light on the NIC port

  3. Verify NIC HW is working properly by using Device Manager and ensure the active NICs are showing green

  4. Verify gateway, IP, subnet mask, DNS, DNS suffixes, etc. are properly configured.

  5. If everything is properly configured and HW is working, you should be able to get a ping response from the gateway.

Check Network Connections
Here are some other checks you should perform to ensure proper network connectivity:

  1. ipconfig /all will display all you TCP/IP settings including you MAC address

  2. ipconfig /flushdns will flush your dns resolver cache

  3. ipconfig/displaydns will display what is in your dns name cache

  4. Netstat -an command will show all the connections & ports from a machine

  5. Nbtstat command will show net bios tcp/ip connection stats

  6. Tracert <IP or DNS Name> command will show you the path the packet takes, the routers, and the response time for each hop.

  7. pathping <IP or DNS Name> command combines ping and tracert to the 100th degree.  It pings each hop 100 times and is great for testing wan connectivity

Disk Space

All kinds of bad stuff can happen when your disk space is filling up.  The best way to alleviate this is to write a script to notify you when you reach a certain threshold. In a future post I”ll share a method for you to do just that…however if there is a problem and you need to perform a health check then here is how you check the space the old fashion way.

To check disk space manually:

  1. Right Click on My Computer

  2. Select Manage

  3. Select Disk Management

  4. Validate each disk more than 10 percent free space

Event Logs

Event logs can reveal a more historical perspective on what is going on with the system and applications. Things to look for when troubleshooting event logs is to query either the system or the application logs and look for the presence of events that have a timestamp near the time of the issue you are troubleshooting.

Events have 3 categories in the event viewer:

  • Informational: Noted with a white icon and letter ‘i’. Successful operations are logged as informational. Usually not used in troubleshooting problems or failures

  • Warning: Noted with a yellow icon and exclamation point. These usually are looked up as they serve as predictive future failure indicators, such as disk space running low, dhcp ip address lease renewal failures, etc.

  • Error: Noted with a red circle icon and ‘x’. These are indications that something has failed outright and are a good starting point for troubleshooting.

When looking at event logs, use the information to determine the following:

  • Is the incident tied to a particular time or outage incident?

  • Is this a one-off, or has this particular error occurred multiple times in the past?

  • Does this error appear on other systems or is it unique to the system that has failed?

Also make sure you take a look at eventcombmt from Microsoft.  This tool allows you to search the logs of multiple machines.  The benefit to this is to see if a specific error or warning message is also occurring on other systems.  This can help rule out issues.

Services

Troubleshooting services should be limited to the specific that is affected by the problem being troubleshot. Each server will have specific services varying upon the types of applications running. You should document how your servers services are configured to and compare that to the server in question to see if anything is not configured correctly.

Cluster

Servers that host applications and services that require high availability should be clustered so that if one node fails the other can pick up the workload.  Clustered servers need the same type of health checks as stand-alone systems except you will want to check on the health of the cluster.

Check Cluster Resource Status

  1. Open Cluster Administrator: Log onto server, select Start –> Run –> cluadmin

  2. Check the Resources and ensure all are Online

  3. If Cluster Administrator does not open, ensure that the Cluster Service is running on the node.

  4. Cluster resource status can also be checked from a remote server. From a command prompt, just type – cluster res <cluster name>

Client Side Health

  1. Right click on My Computer, select Manage

  2. Open Device Manage

  3. Drill down to SCSI and RAID Controllers, verify that the HBA HW is visible and does not show any errors

  4. If it does not show up in Device Manager, you may need to re-scan for the HW, re-seat the fiber card, or re-install the driver.

  5. If the HBA is showing healthy in Device Manager, open the tool that you use to view configuration and settings for the fiber card and verify there aren’t any transmit/receive errors on link statistics or counters

Switch Health

  1. Make sure fiber is properly connected to each switch

  2. Make sure switch has no errors

  3. If you’re using zoning verify it is properly configured

Check Fiber and SAN Connectivity

  1. Log onto san appliance and verify that the SAN is in general good health and no major errors are present for the controllers, loops, switches, or ports.

  2. Ensure that the LUNs are presented to the servers in the cluster

NLBS

Some applications will require you to spread the load across multiple servers.  Web servers are a very popular choice to network load balance.  As with clusters we will need to check the status of the load balancing.

Check NLBS Status CMD Line

  1. From a command prompt on the local system, run ‘wlbs query’. This will give you the convergence status of the local node with the nlbs cluster.

  2. Other useful NLBS commands: wlbs stop (stops nlbs), wlbs start (starts nlbs), wlbs drainstop (drains node)

Check NLBS Configurations

  1. Open up the network properties –> Network Load Balancing, right click & select Properties

  2. On the Cluster Parameters tab, verify that the IP address is configured for the shared NLBS IP and that the subnet mask, domain, and operation mode are configured correct1y.

  3. On the Host Paramters tab, make sure each node of the cluster has a unique host identifier. Also verify the IP and subnet mask are configured for the local values.

  4. Also make sure that your switch has a static ARP entry if using multi-cast NLBS. The entry should be that of the virtual MAC of the cluster. To get the virtual MAC of the cluster, you can run the following command: WLBS IP2MAC <virtual IP address>

Name Resolution

To healthcheck name resolution, open a command prompt and enter the following

  • nslookup <servername>

Verify that the servername is correctly entered in DNS

If a record does not show up in the DNS query, or maps to a different name, perform a reverse lookup by IP address to see what name is associated with the IP address * nslookup <IP address>

If no name shows up associated with the IP address, log into the domain controller and check the DNS records for this particular name/ip address

  1. From a Domain Controller go to start–>run–>dnsmgmt.msc

  2. Expand the Forward Lookup Zones

  3. Expand the zone for you primary zone that holds the records for the system/s you are troubleshooting

Validate that the record exists. If it does not exist manually enter the record name and IP address by right clicking on this same zone,

  1. Select new host (a)

  2. Enter the name and IP address

  3. Check the box next to Create associated pointer (PTR) record

  4. Click add Host

Additionally log back into the node that you manually entered the record for and ensure that DNS is registering in DNS

  1. Right click on the My Network Places icon on the desktop and select Properties

  2. Double click on the primary adapter

  3. Select properties

  4. Highlight internet protocol (TCP/IP) and select properties

  5. Validate the IP addresses of the DNS servers are correct

  6. Select Advanced

  7. Select DNS tab

  8. Make sure the box is checked next to Register this connection’s address in DNS

As I wrap this up I realize there is so much more that can be done.  Each application type of server needs its own set off health checks.  For example web servers, terminal servers and database servers.  Remember this is just the baseline for each server and that other components can and should be layered on top of it.  Again I would love to hear from others so please feel free to add you comments below.

How Active Directory PowerShell CMDLETS find a DC running Active Directory Web Services

If you have been playing with the the AD PowerShell cmdlets you know that it requires a few things to run, first Windows Server 2008 R2 or Windows 7, the .NET Framework 3.5.1 and of course if you want to manage an AD domain you need Active Directory Web Services (ADWS) installed on at least one domain controller. 

By the way ADWS requires TCP port 9389

So how in the world does a Windows 7 system know how to find a DC running ADWS?  Well your client running PowerShell will use the normal DC locator process.  First the client will determine which site it is in nltest /dsgetsite and then it will determine the closest DC nltest /dsgetdc:<FQDN Domain>.  It is looking at the DC for the following flag:

DS_WEB_SERVICE_REQUIRED

More info on that flag can be found here.

Now what if you don’t have Server 2008 R2 DCs?  With Server 2003 and Server 2008 a problem occurs because the Net Logon service of those domain controllers does not recognize the DS_WEB_SERVICE_REQUIRED flag.  There are two hotfixes (one for what ever version of AD you are running) available to fix that in those environments.  Server 2003 and Server 2008

After you install this hotfix the AD PowerShell module and Active Directory Administrative Center will be able to locate DCs that have Active Directory Management Gateway Service installed, similar to Active Directory Web Services (ADWS) on a Windows Server 2008 R2-based computer.

PowerShell Script Center

I’m sure a lot of you have been playing with PowerShell.  If not you better get on it!!!  I’m not as far along as I wish I was but there is help out there.  One great place is to see what others have done.  Microsoft’s TechNet Scripting Center has a place where you can upload your own scripts and search what others have done.  This is great for a community of learning developers…did I just say developers…ewwwww.  :,,)

This link provides a shortcut to filter just the Active Directory related scripts.  From here you can find scripts on Computer Accounts, Domains, Groups, Monitoring, OUs, Searching Active Directory, Sites and Subnets and User Accounts!

If you want to just view all the PowerShell scripts just hit this URL – http://gallery.technet.microsoft.com/ScriptCenter/en-us.  Here you will scripts on Active Directory, Applications, Backup and System Restore, Databases, Desktop Management, Group Policy, Hardware, Interoperability and Migration, Local Account Management, Logs and monitoring, Messaging & Communication, Multimedia, Networking, Office, Operating System, Other Directory Services, Printing, Remote Desktop Services, Scripting Techniques, Security, Servers, Storage, System Center, Using the Internet and Windows Update.  WOW that is a wealth of info.

Enjoy and please share if you have any cool ones yourself.