Word of the day: sucktastic.

Best way to start the month: come in to find a cacaphony of phones ringing, because Nothing Is Working, and that the only one of your minions who's in this early is being phone-guided by the network admin to find out WTF. It turns out that the switch that everything important is plugged into went tits-up at 8:50a, and that was power-cycled at 9:15a (after some distraction from the UPS giving odd errors), which worked, mirabile dictu; unfortunately a hell of a lot of the servers were left in states ranging from "confused" to "dead". Many alerts, reboots, frantic runnings-around, and diagnoses ("ok, so the symlink we have at /etc/nologin has gone walkies, so /etc/nologin now actually DOES exist, that would be why no one can log into p2 or p3") later, everything is back and calmness is restored.

Until the switch burped again while I was trying to type this out the first time. Lather, rinse, repeat.

As soon as Ed gets in he will replace the switch, but this requires him a) getting here from the Bronx, and b) actually building the new one. We have pieces parts, I understand, but they must be melded.

Other things we learned today...
Good Idea: having instructions for putting a system-status message in the voicemail, so people don't bug us.
Bad Idea: having the instructions only online, so you can't get to them if the system is down.

I have this feeling I get no aikido today, either. GODDAMMIT

Haw.

I went to class anyways. And lo, the switch failed not.

However, the CSU/DSU our T3 is plugged into did.

It's a shpedoinkle day!