Introduction

Having a change management lab is an important tool for a network administrator. Whether your network is 50,000 nodes spread across 3 continents or 50 nodes in a single location, a change management lab will allow you to test any potential changes  to your network in a safe and controlled environment. A common and crucial use for a change management lab is to deploy patches and test the effect they will have on your network in the lab. If you've read my whitepaper on using Microsoft SUS you saw that I had problems with a particular hardware configuration rebooting after a set of Windows patches was pushed out, and actually; the same problem has occurred again since then with the same systems. It's a good idea to deploy your patches to test systems that mirror your production environment so that any problems that might arise will not negatively impact end users.

Other changes you need to make may include replacing a firewall, upgrading or changing the OS on your servers, or implementing new controls such as GPOs or file system permissions. These are huge changes that you know have the ability to cause serious problems. When implementing these types of changes to your network your change management lab not only gives you an opportunity to test the effect your changes will have, but also to practice the procedure used to make the changes. Upgrading your Exchange Server to a new version, for instance, is something you want to test, document thoroughly, and practice repeatedly before making a single alteration in the production environment. Yet another use for the lab is to perform penetration testing and vulnerability assessments without having to fear that you will disrupt service. So we can see that a lab will have several critical uses: implementing and testing changes, practicing the procedures involved with implementing major changes, documenting those procedures thoroughly as you step through them, and performing security assessments.

When you look at the types of changes that must be implemented in a network, you can see that at least once a month there is some change that must be made. Microsoft mentions that there are 2 different types of labs: ad hoc labs, which are built for a specific project and torn down when it is complete; and change management labs, which function continuously. Given that our jobs as network administrators will nearly always require the use of a lab this article will only cover change management labs.

Planning & Design Methodology

Now we know why every systems administrator should have a change management lab at their disposal. Two main questions immediately follow. First, what is necessary in the change management lab in terms of hardware, software configurations, and networking equipment and setup? Secondly, how do we effectively use the change management lab? There are some other questions as well in terms of how much administration work will doubling your network incur? What is the cost, and how do you budget for the lab?

This is starting to seem like a big headache, right? It is key to remember that the lab does not need to be massive, and that its purpose is to assist you in the administration of the production environment. The headache you are eliminating with the lab is the uncertainty of how the big firewall replacement will go on Saturday, finding out that the server you are planning to run NTop on is not powerful enough to also run BASE before you go live with it. As far as cost goes, we must simply use whatever budget is available. If you do not have the money to purchase VMWare and high-powered servers capable of running dozens of virtual machines then you don't look at that as possibility. Work with what you've got.

Your lab design will be based upon 3 different components of your environment: the hardware, the software, and the network. You will need an inventory of everything that exists on your production environment within each of these 3 categories. There are many tools you can use to generate this inventory. If you have Microsoft Systems Management Server then you already have a wealth of tools to do the job. If not, then you can always use scripting in a *nix or Windows environment to do the job. This article is not going to cover the use of SMS or scripting though, so you'll need to learn those on your own. The important thing is that you will need to have an inventory of hardware, software, and network services, and that there are tools out there to help you generate these inventories.

You should really try to have at least 1 hardware config in the lab that represents an identical or similar config that is in production. If you use 10 Dell Optiplex 150s in production then make sure you have at least one of them in your lab. This is going to ensure that whenever you are making changes to software (like rolling out new software or applying patches) you will catch any potential problems in the lab. If your new accounting software is running sluggishly on a certain hardware config in the lab despite it meeting the minimum recommended requirements, then you can make adjustments and avoid problems in production. The way to approach your hardware is to try to break things into different models. You may have 20 computers that are all 1 model, but certain individual systems have more RAM than others, or a CD-RW drive while others do not. These differences are somewhat insignificant for our purposes. Focus on representing each model in the lab, not each slightly unique configuration.

Our goal is to give a good representation of the production network in the lab and in most cases a slightly different model of CD drive is not going to make a difference in our testing. Memory might have an effect, but there is still no need to include Model A with 128MB of RAM in the lab as well as Model A with 256MB if they are both the same model of computer. Being this specific in our mirroring is going to result in having a lab network as large as the production environment and incurring too much administrative work for it to be practical. In these cases we will use the system with only 128MB of RAM in the lab. The reasoning here is that if any problems would occur on the system with 256MB of RAM they should also occur on the 128MB RAM system, but the opposite is not likely to be true. So we will be able to recognize all the problems that may occur within that model of machine.

Software you use in production should also be installed in your lab. Just as we want to represent our production hardware in the lab we want to represent our production software in the lab too. This means installing the OS versions in the lab that you use in production, the same software packages and versions used in production. Now, if you run Windows 2000 on some Model A computers and Fedora Core 3 on some other Model A computers you may need to go against what I've stated in the last paragraph, and use 2 of the same model computers in the lab. You are doing it because of the need to accurately represent software in the lab in this example.

To this point, you've created an inventory of hardware from your production environment. You also have a software inventory from production which you will mirror in the lab. Now you should inventory the network services you are running in production. You need to have a good list of what machines in production are running DNS, NIS, WINS, DHCP, Active Directory, Email, VPN, and other services. Also, you will need to know what software is providing these services. Is your DNS BIND version 5, BIND version 8, or Windows Server 2003 DNS?

The important thing as far as networking is concerned is to try to mirror the logical layout of the production environment as it occurs behind the firewall. If you use VLANs in production then use similar VLANs in the lab. Your firewall should have the exact same rule set. Most importantly, you should mirror your directory infrastructure. If you use Microsoft AD, then you should have a DC in your lab representing each DC in production, and the DCs should contain the same data as production at the beginning including all users, groups, GPOs, login scripts, etc. Keeping the same IP addressing scheme is unnecessary, but you should keep the same general network structure- a DMZ, LAN, and WAN in production should be reflected by a DMZ, LAN, and WAN in the lab if possible.

Also, you may end up consolidating servers in the lab. You may need to run email, 2 Windows domain controllers, DNS, DHCP, WINS, and a file and print server on 3 servers in the lab where these jobs were spread among 7 servers in production. This could be problematic, but does not have to be. You are testing for when problems arise in the lab so you can eliminate or prevent the problem from occurring in production. Microsoft suggests that you do NOT install Exchange Server on a domain controller, and one thing we do not want to do in the lab is artificially manufacture problems we will never see in production. Therefore, in our situation we should split DNS, DHCP, WINS, and 2 domain controller roles between 2 of our lab servers. On our third lab server set up Exchange. In this way we have done our best to accurately represent the production environment in our lab without introducing conflicts or problems artificially, but while keeping the number of machines in the lab to a minimum to reduce the administrative overhead the lab will incur.