the nuclear option
Yesterday morning I figured that I'd upgrade the last (and primary) computer at home from Fedora 8 to Fedora 9. While the other upgrades weren't perfect, they weren't disasters either. This upgrade was a complete & utter failure. While the upgrade itself appeared to go very smoothly, the first boot showed immediate signs of problems. First, it was still using the Fedora 8 kernel. Second, numerous services failed to start due to missing libraries (libnssutil3.so, libplds4.so, libplc4.so, libnspr4.so). After the boot process finished, I discovered that rpm was completely broken. And by broken, I mean that the entire RPM database was gone. To make matters worse, the rpm binary was silently failing to do everything. Running a "rpm -ivh foo.rpm" immediately exited with no output whatsoever. Even a simpmle "rpm --initdb" exited immediately with no output. I tried to strace the process, and ended up with 4958 lines of output, with hundreds of lines of missing libraries. I copied some of the missing libraries over from Denise's laptop in the hopes that this would get things into a state that was good enough to re-initialize the RPM database, but no luck. RPM remained broken, utterly. I thought that maybe I'd run the upgrade again (off of the DVD), but just attempted to upgrade thunderbird, and then claimed it was finished. Worse, now the OS no longer booted, and just hung at "loading GRUB stage 2...".
At this point I knew that I was completely screwed. I booted off the DVD into rescue mode, and figured that I'd back up all my important data and just do a fresh install of Fedora 9. Hardly the ideal solution, but it should have gotten me back into a relatively useful state. I had about 65GB free on an unimportant partition on one of my SATA disks, and figured that I'd just put everything in there. Sadly, I ended up having over 100GB of additional data that I needed to save, which meant I grabbed an old 250GB IDE disk, and tried to use it. It was fine for about 15GB of data, and then started spewing ATA errors, and completely died. At this point, it was mid-afternoon, and I ended up having to selectively delete stuff to free up sufficient space to save the data that I absolutely couldn't afford to lose. That took another two hours, and finally it was done.
It was nearly 4PM, and I performed a fresh install of Fedora 9 without any issues. It booted ok, and I spent the rest of the day and evening getting everything updated and restoring all the data. Roughly 24 hours after starting this fiasco, everything is back up, and the Fedora upgrade is complete.
Needless to say, I'm fairly pissed off at Fedora. 9 has been nothing but problems. In hindsight, I should have just gone with a yum upgrade of everything, as that would have been far less likely to break the rpm database.