« Deploying SQL Server 2000 onto Windows 2003 clusters with more than 4 nodes | Main | eWEEK.com Podcast: Automate Mundane Database Tasks to Increase Value »
How do I know when to patch or to leave well enough alone?
By Eric Gross | July 23, 2008
Every business wants the same thing: a database that services clients with the utmost performance and is always available. In the war against downtime, patches can be a rogue force. When something goes wrong, patches may let instability—the enemy—in through the front door. Patches always promise to fix a problem. If not for that promise, there would be no reason to let such an untested piece of code into your environment. Before you let any patch into your camp, you need to make sure it won’t have a malicious impact or any other negative effects. It makes sense to initially let it only into a bastion where you can analyze its every impact. Patches should always be held to the standard of providing a positive result—not creating any negative effects which will outweigh the intended result. If possible, businesses should always wait for the subsequent patchset, which normally contains all recommended patches made public since the last release. If it’s not feasible to wait that long, a patch bundle may be an alternative since bundles can be more widely deployed and tested.
Whenever a patch presents itself, you should vet it just as you would challenge any other change to a working system. Sometimes patches turn into candidates for installation based on a clear-cut symptom a specific patch has been issued to address. Other times the confidence that a particular patch will fix a symptom is lower, and sometimes, it can be a shot in the dark. It is always prudent to search the web for patch-related issues specifically related to that patch. Beyond that, take advantage of the limitless number of service requests available to you to confirm all implications of using this patch. You may find during your interaction with product support that you are missing a prerequisite for patch installation. This approach makes even more sense, because Critical Patch Updates clutter the landscape in different ways based on the releases you are running. No matter how much confidence you have that a patch will remedy an issue, you should always test it in an environment that is as similar as possible to the production case. A point patch can break seemingly unrelated things, so testing needs to include a full regression suite (in addition to the obvious and critical confirmation that the patch fixes its intended symptom).
Patching is one of the endless reasons that you should minimize the number of configurations in play at any one time. Since you’ll need to test each candidate environment for each patch, you should—if possible—reduce the number of unique configurations. Once you have decided to apply a patch, the next step is creating a procedure that you’ll use to deploy and rollback this patch. Ideally this script will be completely automated so that one command takes care of the entire apply/rollback use case: stop dependent processes, backup, apply patch, and restart processes. All of this should be performed in addition to your custom requirements—those related to change control or regulatory requirements. Each part of the procedure that is manually executed increases the probability of failure and inconsistent results. Equally important to the application process is the rollback process. This can be required without notice and in critical situations where unattended work is far preferable for its speed and reliability. Backups are a fundamental part of most rollback procedures, so you should be confident in your ability to restore to a working state.
Remember: Once a patch is let loose, it is important that it be monitored. This crucial final step ensures that the intended results are realized and any negative effects are tracked and potentially resolved.
Topics: Database Automation, Patches