09:54 Thursday, November 11 2007

kernel panic!

I (thought that I) had some time to kill yesterday afternoon, and went ahead and upgraded from Fedora 7 to Fedora 8 via yum. Everything appeared to go well, until I rebooted and ended up with a kernel panic:

/bin/nash: /lib/libc.so.6: version 'GLIBC_2.7' not found (required by /lib/libcrypto.so.6)

Errr, this was not good to put it mildly. At first I was thinking that somehow glibc never got updated. I booted off the Fedora 8 DVD into rescue mode, and chrooted into the filesystem to poke around. Nope, glibc definitely had been upgraded. So was nash, and so was openssl (from which libcrypto originates). So, at this point I'm very confused, and a bit panicked.
One oddity that I did notice was that nash should have been in /sbin rather than /bin, and then it hit me, this was the initrd that was broken. But I'm still not sure exactly how or why. I grab the initrd, and extract its contents:

gunzip < initrd-2.6.23.1-42.fc8.img | cpio -i --make-directories

and start poking around. I confirm that yes, nash is in /bin. I look at /lib/libc.so.6 and see that its symlinked to a /lib/i686/nosegneg/libc-2.5.so. Aha! I have no clue where /lib/i686/nosegneg/libc-2.5.so came from, but then realize that I have it on the master filesystem too. I'm still not entirely sure why mkinitrd is using it instead of the libc.so in /lib on the master filesystem. Anyway, I grab /lib/libc-2.7.so and dump it into /lib/i686/nosegneg (inside the initrd filesystem), and fix the symlink to point to it instead of libc-2.5.so. While doing this, I noticed that libm.so.6 is also pointing to an older GLIBC_2.5 version, so I fix it in the same fashion. I then repackaged the initrd:

find . | cpio -H newc -o > ../initrd.cpio 
gzip -9 ../initrd.cpio

And reboot. Bingo, no more problems. At this point, I fixed all the same files under /lib/i686/nosegneg on the master filesystem so that future generated initrds wouldn't be similarly broken. What I still don't understand is why this got broken originally, or what this stuff inside of /lib/i686/nosegneg actually is. None of it appears to be part of an RPM.

</p>