Friday, March 19, 2010

Fixing a broken AD Domain (part 3)

In part 1 and part 2 I explained what has happened to my AD domain, and the steps I’ve taken to fix it.

Now, let’s get a better safety net in place!

I’ve got a good, reliable PowerEdge 2650 that was donated by another IT guy (thanks Jim!) It’s been humming right along, waiting for something to break an opportunity to take on some further roles.

My PE2650 is a Windows Server 2003 R2 box. ‘Dorothy’ is a tired old Windows Server 2003 SP2 box (as are my other servers). To make the PE2650 a domain controller, I need to:

  1. Extend the AD Schema to accomodate 2003 R2’s new functionality
  2. Add the PE2650 as a Domain Controller

Step 1: Extend the schema

  • Log onto ‘Dorothy’ as a schema admin
  • Use this excellent TechNet article to extend the schema. Read carefully and follow every step
    note: I had no issues extending the schema, and the directions were spot-on. Why re-invent the wheel?

Step 2: Add the PE2650 as a Domain Controller

  • Log onto the PE2650 (w/ admin rights)
  • Follow this excellent Petri article to add the PE2650 as a domain controller
    note: I did not have to restart the netlogon service

Fixing a broken AD Domain (Part 2)

Earlier today I started a series about fixing my problematic AD Domain. This is part 2

So, at this point I have an unhealthy Active Directory infrastructure. What I’m facing:

  • Domain Controller that doesn’t exist
  • DHCP server that doesn’t exist
  • Working DHCP server that’s worrisome
  • The need for a 2nd quality Domain Controller and backup DHCP server

What I’ll be tackling in this post:

  1. Removal of the bad Domain Controller (DC) from our systems
  2. Removal of a DHCP server that was added during RIS trials (the trials have been canned)

Step 1: Verify documentation of AD servers

Be sure to include the following:

  • Domain Controller’s (DC’s)
  • Schema Master
  • Domain Role owner
  • PDC Role
  • RIP Pool Manager
  • Infrastructure Owner

In my case, all of these roles belong to my primary server, ‘dorothy’

Also make sure to document any Trust relationships (other domain’s, etc.). I don’t have any

Step 2: Remove the failed Domain Controller

To remove a failed/dead domain controller, I used the following method:

  1. At a command, type ‘ntdsutil’ and hit enter to open the Directory Services Utilities menu
  2. Type in ‘metadata cleanup’ and hit enter to enter the metadata cleanup menu. This will help us clear out the stale information referencing our dead DC.
  3. Type in ‘select operation target’ and hit enter
  4. Type ‘list domains’ and hit enter. This will list the domains we have available. Mine is called ‘’ (no we don’t own it; it’s a long story)
  5. Type ‘select domain 0’  and hit enter ( i.e. DC=calvary,DC=com is 0 for me)
  6. Type ‘list sites’ and hit enter. This lists all sites. I only have one
  7. Type ‘select site 0’ and hit enter (my site # is also 0)
  8. Type ‘list servers in site’ and hit enter. This shows you the list of servers in the site (domain controllers). I took note of the # for my bad DC (1)
  9. Type ‘select server 1’ and hit enter. This selects the bad DC
  10. Type ‘q’ and hit enter. We need to go back to the metadata cleanup menu to finish
  11. Type ‘remove selected server’ and press enter
  12. I got a warning message asking for confirmation. Obviously I wanted to complete this (because the physical DC doesn’t exist anymore)
  13. Now you get a confirmation line
  14. Type ‘quit’ and hit enter

Next, I need to remove the DC from the corresponding other areas: Active Directory Sites and Services, Active Directory Users and Computers, DNS, and possibly DHCP

  1. Open up ‘Active Directory Sites and Services’
  2. Expand the site that the DC exists in
  3. Right-click on the bad DC, and then left-click on ‘Delete’
  4. Open up ‘Active Directory Users and Computers’
  5. Open the ‘Domain Controllers’ container
  6. Delete the bad DC computer object. You may get a warning. Heed it and proceed
  7. Open the ‘DNS’ snap-in
  8. Remove the bad DC records (CNAME, hostname, NS, A, etc.) from the appropriate Forward Lookup Zones. In my case, I had 2 areas to go through and check:
  9. Remove the bad DC records from any Reverse Lookup Zones. In my case these were already clean (I’m unsure as to why)
  10. Go through and check DHCP to make sure that you’ve removed all traces of the bad DC. I didn’t have anything in DHCP, but wanted to double-check. If the bad DC was a time server or some other role, make sure you make the proper modifications

I went ahead and restarted DHCP, flushed my DNS server’s cache, and then restarted DNS.

Great! Now I have cleaned up my AD infrastructure.

Let’s get the ‘extra’ DHCP server that was added during RIS trials removed
note: the DHCP server for RIS never went active. It was added to the DHCP server list as a prerequisite

  1. Open up the DHCP snap-in on your DHCP server (‘dorothy’ for me)
  2. Right-click on the icon labeled ‘DHCP’ (not the icon for ‘dorothy’)
  3. Left-click on ‘Manage authorized servers…’
  4. Select the DHCP server that doesn’t exist and then click ‘Unauthorize’

That’s it!

Up next time:

  • Extending the schema of my AD domain to support Windows 2003 R2
  • running adprep on ‘dorothy’
  • Adding a 2nd (replica) domain controller

Fixing a broken AD Domain (Part 1)

NOTE: This post (like many) is mostly for my documentation. If you derive some value from it, great!

I received a phone call on the way in to work today that we had the following symptoms:

  • Some staff weren’t seeing their network shares
  • Some staff had issues logging on
  • The internet was working sporadically
  • Some staff noticed no difference

Needless to say, this caused me to breathe quickly for a few seconds, and then I started praying on the way in, because it means that:

  • Domain controller(s) aren’t responding
  • Our internet connection (Bonded T-1’s) are down
  • Our firewall is having issues
  • Some other, unknown, time-consuming thing has happened

After praying, I remembered Romans 12:2 ; this was another reminder that God has bigger plans for my life, because this verse was brought up during a morning accountability group this very morning. Set my mind on Him and His things, and He will hold true to His promises.

Upon arriving at work, I found the following:

  • My FSMO, PDC, GC, and otherwise-depended-upon server was having serious issues. NTFRS, NTDS, Directory Services, DHCP, DNS, etc. were all throwing out errors.
  • My firewall (pfSense) was showing connectivity issues on the WAN link
  • Staff was able to log on with cached credentials, but could not access network resources
  • Staff that had previously logged on (before 8:45am) were sometimes working normally
  • My bosses laptop OS decided to finally junk out. It’s been showing bad signs, but ‘it’s not at the top of the list’

What I did:

  1. Checked the backups of my FSMO/GC/Schema Master/RID/Infrastructure server. We’ll call her ‘dorothy’
  2. Rebooted Dorothy

What happened:

  1. Staff was able to login
  2. Network resources mapped and were available to staff

Whew! Crisis averted, right?

WRONG! This was just a small band-aid on a gushing wound. Time for surgery.

What the real problem was:

  • We have a ‘stale’ Domain controller (died)
  • ‘Dorothy’ is a very old server (at least 6 years old). She’s also had an in-place motherboard replacement done (to dis-similar hardware).
  • DNS is flaky
  • DHCP is showing signs of flakiness on Dorothy

Up next: Fixing all the issues. Hopefully I’ll be done today

Wednesday, March 10, 2010

Dell Desktop System Software – Do you use it?

Recently I’ve been working on updating our images for some new (to us) machines we purchased and our Office 2007 rollout. Along the way, I noticed something interesting:

Dell recommends that you install ‘Dell Desktop System Software’ before installing any drivers, etc. on your new Windows installation. This is the description of DSS:

Desktop System Software (DSS) is a utility that provides
critical updates and patches for your operating system

I did a little more digging, and here’s an interesting tidbit I found on a messaging board:

This is the equivelent of windows update, but it updates dell device
drivers, and dell supplied software. MS doesnt do that.

I haven’t been using Dell DSS for my machines, and I haven’t had any serious recurring issues. I did have one issue, one time, with 2 Optiplex 755’s needing a BIOS update, but I determined that after calling support. I wonder now if this would have made that process easier or automated.


What’s your experience? Do you use Dell DSS? Has it saved you a support headache? Has it caused a support headache?