Initrd: The Pivoting Moment

The other day a system had its /boot partition wiped clean. That system had Centos 5.5 running on it, so we decided we could just copy the content of /boot

from one of its sister machines. The machine had a few problems… The first problem was the GRand Unified Bootloader. There must be something I don’t quite understand about GRUB, but we had to reinstall grub in to the MBR. Pretty simple procedure which I will wrote seperately on.

The other problem was when I booted the machine up it couldn’t pivot_root to /newroot which reminded me of two things. First I hate initrds and secondly that Centos and Red Hat both use them. Before I begin with the method I used

to fix it, let me explain what the initrd provides your system.

Here is the basic problem. Your kernel is loading but; in order to hand off the boot sequence to init the kernel needs to read some information off your disk but; your kernel doesn’t have the module compiled in to itself. The solution, besides recompiling the kernel with the module built in, is to

boot to a temporary root file system that the kernel does have built in support for. This temporary root file system will have all the commands and kernel modules required to mount the real root file system and pivot the current root file system with the new root filesystem.

There are some other cool things you can do with initrds. For example you can initiate the frame buffer and provide a console theme very early on in the boot sequence for providing eye candy on desktops. Some volume encrypted techniques rely on initrd to provide just enough resources to decrypt and root file system. I’m sure there are any other useful uses for initrd, but that’s beyond the scope of this post.

So that said, the problem we’re trying to solve is to boot a kernel that has drives necessary to mount the root file system built as external modules. Luckily there is a script to build them for you. The problem is that we’re not in a catch 22. My solution was to simply boot a Gentoo Live CD which should have and load all the necessary modules to access the Centos installation. By default Centos makes use of LVM. I had to activate the root filesystem to mount it. The command is `vgchange -ay` and that will activate all LVM volumes which will appear in /dev/mapper and the root volume for me was VolGroup00-LogVol00. I simply mounted that `mount /dev/mapper/VolGroup00-LogVol00 /mnt/gentoo` the destination is arbitrary. You’ll also want to mount your /boot partition, if /boot doesn’t reside on the root file system.

Now our root file system is mounted so we can access the Centos installation and make use of the kernel modules and initrd creation script. Before we begin, it’s important to make sure the dev, proc and sys file systems are mounted. There are numerous ways to do this, but the easiest way is to bind them to our destinations.

~# mount -obind /dev /mnt/gentoo/dev
~# mount -obind /proc /mnt/gentoo/proc
~# mount -obind /sys /mnt/gentoo/sys

The reason this is done is because the initrd creation script probes various system information to find out what hardware modules must be loaded in order to boot the system. That information is available from various parts of the aforementioned file systems. Now

we simply need to change root to the Centos installation. You may experience problems doing this so I will provide you with, what I hope is, a sure fire backup command. If this command fails `chroot /mnt/gentoo /bin/bash` because the library paths are messed up, try running `chroot /mnt/gentoo /bin/sh` which should have less library restrictions if not statically compiled. It should now appear that you are in your Centos installation. It’s time to make the initrd file.

To make the initrd you must know the kernel version. To figure that out use this command `cat /boot/grub/grub.conf`. In my case we’re using Centos 2.6.18-92.1.10.el5. The kernel would be called vmlinuz-2.6.18-92.1.10.el5 . You can confirm this by running `ls -al /lib/modules`. To create the initrd run the following command

~# mkinitrd /boot/initrd-2.6.18-92.1.10.el5.img 2.6.18-92.1.10.el5

The second argument is the file that it will try to make. In my case it already existed because we’d copied the /boot contents from another host. I had to rename the file and run the command again. The second argument to the mkinitrd is the version of your kernel and it directly correlates to a directory in /lib/modules so if it doesn’t exist there then you need to check your work or that kernel wasn’t installed on the Centos version. If that is the case try to find an older kernel that was installed. Failing that you may have to copy that directory to the machine in question as well.

You should be set and you should be able to boot.

Segmentation Fault: My CUPS Runneth Over

Perhaps you’ve had a recent

problem with CUPS and Mozilla. I don’t print often and I recently updated my system’s packages. Today I tried to print something from Thunderbird for the first time in years. Thunderbird crashed so I tried it on Mozilla to print from webmail and to my <sarcasm>amazement</sarcasm> it also crashed! I started investigating and I discovered that it was segmentation faulting. I did some research and I found that Apple’s CUPS version

1.4.4 and Mozilla do not get along. There is a resolved CUPS bug for this problem. They basically isolate the problem to the way Mozilla is opening the CUPS library. They say there are two possible solutions, which don’t involve CUPS. Mozilla needs to change the way they’re dlopen() and dlclose() of CUPS or get OpenSSL to fully support threading.

The problem is that the Mozilla apps are dlopen’ing libcups, which then initializes the SSL library. They then dlclose libcups after the print dialog goes away which leaves the OpenSSL threading stuff pointing at functions that are no longer in the process address space.

One of two things needs to happen – Firefox/Thunderbird need to stop using dlopen/dlclose (or at least dlclose) for libcups, or OpenSSL and GNU TLS need to actually support threading out of the box and not depend on the application or library to provide threading support.

The nice guys at CUPS now leave us in the hands of third parties to fix the problem. Some solutions I’ve read say to downgrade CUPS to 1.4.3 which apparently doesn’t exhibit this bug, which isn’t CUPS’ introduction. I wasn’t exactly satisfied with this. If you read in to the wording you will realize that there are three solutions. I mean… dlopen() isn’t an SSL command and dlopen() doesn’t provide dependency support. Then who is loading libSSL? CUPS… So if the problem isn’t CUPS, then compiling CUPS without SSL support shouldn’t fix the problem. For you binary guys, have fun… For you non binary guys keep reading…

The configure script for CUPS allows for the use of –disable-ssl. Setting this will fix your problem. Compile then like normal. Keep in mind that this will disable SSL for the CUPS daemon. You wouldn’t be able to do secure printing over the network or provide an encrypted web front end. For most users we use CUPS to just print and by default it isn’t listening on public interfaces.

If you’re using Gentoo then the following should work fine for you…

# echo “net-print/cups -ssl” >> /etc/portage/package.use
# emerge -av cups

Before I start any flame wars, I am not saying that Mozilla is correct. I’m especially not saying this because I haven’t run in to any other applications with this problem. I am saying that the

CUPS developers are diluting the truth about the problem and not exposing a possible solution for people having difficulties. If 1.4.4 has bug fixes, features or hardware support that is required then the user may not be able to downgrade or wait for a third party solution. Compromises are occasionally necessary and CUPS shouldn’t be run in the same manner that Apple is.

Downloading PHP: Not Exactly What You Were Expecting…

This was an interesting problem I just encountered. While moving a virtual host from an older Apache server to a more modern Apache server the customer’s site on the new server wanted us to download all the htm files. On initial inspection the first oddity I discovered was that they were registering .htm and .html files with the x-httpd-php mod_php module in their .htaccess file.

AddType application/x-httpd-php .html .htm

After over coming the initial wave

of announce I started checking the obvious things. First I made sure php was working at all. Because .php files were loading I knew that mod_php was loaded for this virtual host. I checked the handler for it to make certain that x-httpd-php was being used. I also tried x-httpd-php5. Upon downloading the files I noticed that they were unprocessed by php. Lastly I checked the AllowOverride directive, only finding that our server allows all overrides.

Finally I stumbled on to the problem. There already was an AddType for .html and .htm files along with .shtml .shtm files. The mod_include, the server side include or ssi for short, module was registering them and Apache was not replacing that with what was in the vhost’s .htaccess file. The first solution to the problem was to remove the .htm and .html files from mod_include. The problem with my first solution was that if anyone was using mod_include and using .htm and .html files they would just stop working.

My final solution to the problem was to RemoveType .htm and .html before AddType .htm and .html . Keep in mind that most browsers cache the file and the mime type. I had to clear my browser cache before I was successfully able to view the site. Below is the final configuration.

RemoveType .htm .html
AddType application/x-httpd-php .html .htm