Weekend Power Surge Causes Service Disruptions

At about 9:00 PM Saturday evening, several components in UHB experienced either complete failure or went into a degraded state. At about the same time, a UPS unit in the HSB datacenter also shut down. The most likely cause of this was a lightning strike somewhere nearby that sent a power surge or power drop through the campus power grid, knocking some systems offline. There was enough redundancy in most systems so that this event did not affect access to campus network services. All core campus services such as e-mail, eDocs, DHCP, DNS, and others remained up and functioning. However, Exchange public folders are not redundant and the server in UHB hosting them was knocked out (which also prevented the daily campus announcements from running). Some network ports in the UHB building were also knocked out, as was a print server in the HSB datacenter.

As of now, the affected servers and switches have been rebooted and all but one switch is back to normal operations. That one switch is part of the core backbone network and there are redundant network paths in place so that this is no service disruption.

My thanks to everyone who played a part in getting things working again this morning!


Leave a Reply