Battery Backup

We've recently had a rash of UPS problems that confused us for a while. We have frequent blackouts or brownouts (more than I noticed before I got into IT) and it has caused some interesting moments. Usually a UPS will allow us to go through these short energy spikes without issue. Since our phone systems and data systems obviously need energy, and the world stops when communications stop, we try to keep these systems running for 15 to 30 minutes without power supplied from the outside world.

After a few recent events, we've found that our mega UPS systems don't stay up for more than 10 minutes and will not return to normal operations after power returns. Shortly thereafter, a strong burning smell permeates the data closet. Under closer inspection we were alarmed to find that one or more battery packs are very hot to the touch! As we removed the battery packs we found that they have begun to expand and what seems to be a chemical reaction is occurring where they leak.

As we've removed these batteries and replaced them the system will go back to functioning normally. So what causes this? Apparently after 3 - 5 years of use UPS batteries are prone to losing their ability to hold a charge. Once they do not hold a charge the UPS continues to try to charge them up, causing a very dangerous situation to occur inside the battery. Some reports have been given where they actually start fires.

So what can you do to monitor or prevent this? It seems that there is little to be done other than monitor and replace your UPS systems that are within this time range. We've evaluating the prospect of putting in thermal sensors to alert us to a battery pack that is getting hot. It amazes me that the one system you purchase to protect your electronic investments can ultimately provide their demise.
(Update 16May2011: We have measured the tempurature on some of the failing batteries (like today's) at over 160 degrees. Please use caution when removing these battery packs as they weigh over 40 pounds, are very hot and most likely are leaking acidic liquid when in this state.)