17:54 Wednesday, February 02 2009

14 hours

Yesterday was a fairly decent day until just before 4pm when it all went to hell. First, a coworker ignorantly trashed a test system (NFS mounting something on top of / is just plain stupid). Then the primary NFS server locked up. Upon investigating, we found that it had a cryptic kernel panic. I hit reset, and the 2TB RAID5 array refused to mount, as it couldn't the superblock. Aieeeeeeee. At that point, it was time for xfs_repair. Since I couldn't get the array to mount, I had to call xfs_repair with the -L option (which zeroes out the journal/log), and hope for the best. After nearly 2 hours, and quite a few cleared inodes, it completed. The array mounted, and appeared to be intact. Of course, then I had the long wait for the array to rebuild, which took nearly another 3 hours. All of this meant that I came into work yesterday before 7AM, and didn't go home until almost 9PM.