Have grub2 boot 32 and 64 bit Ubuntu

I just installed a system like this:

ubuntu@ubuntu:~$ ls -l /dev/disk/by-label/
total 0
lrwxrwxrwx 1 root root 10 2012-01-22 18:49 Boot -> ../../sda1
lrwxrwxrwx 1 root root 10 2012-01-22 18:49 ubuntu32 -> ../../sda2
lrwxrwxrwx 1 root root 10 2012-01-22 18:28 ubuntu64 -> ../../sda3
lrwxrwxrwx 1 root root 10 2012-01-22 18:49 Home -> ../../sda5

ubuntu@ubuntu:~$ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 2012-01-22 20:55 3582d70f-f4a5-484c-b14c-45cd740346b9 -> ../../sda1
lrwxrwxrwx 1 root root 10 2012-01-22 20:55 741182a8-3f15-4dfd-994d-654c8a57a9e4 -> ../../sda2
lrwxrwxrwx 1 root root 10 2012-01-22 20:55 1c415472-a770-4d76-be9f-27b8c1408e2a -> ../../sda3
lrwxrwxrwx 1 root root 10 2012-01-22 20:55 3515d523-72a2-4e04-b7da-cb6a1fd572ef -> ../../sda5
lrwxrwxrwx 1 root root 10 2012-01-22 20:55 f1f1cd7e-30cb-44e7-9ef6-986a589e0045 -> ../../sda6

I need 32 and 64 bit separate so I can test driver performance on each, the Home directory is shared and each version of ubuntu is mounted as /ubuntuXX under the other version. /boot also points to /dev/sda1 on both.

When I run sudo update-grub from ubuntu32 it runs fine but I get an error booting ubuntu64. init fails and I'm assuming it is because of the wrong bit type. Shouldn't grub's OS probe be bit-aware? How can I get these two to boot properly.

I ran grub-customizer from a live disk, chose /dev/sda3 as root, selected all partitions in the second step, and then removed all entires except for OS-Prober and memtest. What resulted is the attached grub.cfg. It now lists /dev/sda2 as the only OS option. The grub.cfg has the root UUID set to /boot for every entry.

grub.cfg: http://pastebin.com/Dkdxian4
fstab (both): http://pastebin.com/3sUabYRY

I know grub2 is all auto-generated and menus and jazz, but how do I just nuke all that and manually add entires (I'm not keeping this install long so there's no worry about kernel updates)

I took a shot at bilndly replicating the /dev/sda2 entry but adjusting it for /dev/sda3 however it didn't go well. I did however, manage to get the same error as my first attempt (it's the line right before the kernel panic, run-init)

updated part of grub.cfg: http://pastebin.com/DvfBhrBF

(null)
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom: ... done.
[    3.402989] request_module: runaway loop modprobe binfmt-464c
run-init: /sbin/init: Exec format error
[    3.406303] Kernel panic - not syncing: Attempted to kill init!
[    3.406394] Pid: 1, comm: run-init Not tainted 3.0.0-15-generic #25-Ubuntu
[    3.408290] [<c151a922>] ? printk+0x2d/0x2f
[    3.408338] [<c151a800>] panic+0x5c/0x151
[    3.408388] [<c104b564>] forget_original_parent+0x1e4/0x1f0
[    3.408440] [<c10d3a48>] ? perf_cgroup_attach_task+0x20/0x20
[    3.408489] [<c104b583>] exit_notify+0x13/0x140
[    3.408536] [<c104bd8d>] do_exit+0x1ad/0x3a8
[    3.408585] [<c1313850>] ? tty_write+0x228/0x228
[    3.408632] [<c104c098>] sys_exit+0x18/0x28
[    3.408680] [<c152e424>] syscall_call+0x7/0xb

I'm beginning to think it has to do with the kernel images that are in the /boot directory.

Ok, now I'm certain that it is due to the fact that one partition is 32-bit and one is 64-bit.

Figure 1: exert from /boot/grub/grub.cfg:

linux /vmlinuz-3.0.0-12-generic root=/dev/sda2
initrd /initrd.img-3.0.0-12-generic

Figure 2: Listings of /ubuntu64/vm* and /ubuntu32/vm*

me@GAMMA:~$ ls -l /vm*
lrwxrwxrwx 1 root root 29 2012-01-23 20:41 /vmlinuz -> boot/vmlinuz-3.0.0-15-generic
lrwxrwxrwx 1 root root 29 2012-01-22 13:05 /vmlinuz.old -> boot/vmlinuz-3.0.0-12-generic
me@GAMMA:~$ ls -l /ubuntu32/vm*
lrwxrwxrwx 1 root root 29 2012-01-22 12:41 /ubuntu32/vmlinuz -> boot/vmlinuz-3.0.0-15-generic
lrwxrwxrwx 1 root root 29 2012-01-22 12:22 /ubuntu32/vmlinuz.old -> boot/vmlinuz-3.0.0-12-generic

Figure 3: Magic File Type of /boot/vmlinuz-3.0.0-12-generic

/boot/vmlinuz-3.0.0-12-generic: Linux kernel x86 boot executable bzImage, version 3.0.0-12-generic (buildd@creste, RO-rootFS, root_dev 0x801, swap_dev 0x4, Normal VGA
/boot/vmlinuz-3.0.0-15-generic: Linux kernel x86 boot executable bzImage, version 3.0.0-15-generic (buildd@creste, RO-rootFS, root_dev 0x801, swap_dev 0x4, Normal VGA

Now here's the real kicker:

me@GAMMA:~$ sudo find / -name "vmlinuz*"
/boot/vmlinuz-3.0.0-12-generic
/boot/vmlinuz-3.0.0-15-generic
/vmlinuz.old
/vmlinuz
/ubuntu32/vmlinuz.old
/ubuntu32/vmlinuz

That's the only kernel on my system! How is that even possible? I'm running 64-bit right now (/dev/sda3 is / and uname reports 64-bit)!

I checked the package contents on packages.ubuntu.com and the am64 version of linux-image-3.0.0.15-generic lists /boot/vmlinuz-3.0.0-15-generic as a file, so I ran the following:

me@GAMMA:~$ sudo openssl dgst -md5 /boot/vmlinuz-3.0.0-15-generic > ~/3.0.0.15x86
me@GAMMA:~$ sudo apt-get install --reinstall linux-image-3.0.0-15-generic
(Output Omitted)
me@GAMMA:~$ sudo openssl dgst -md5 /boot/vmlinuz-3.0.0-15-generic
MD5(/boot/vmlinuz-3.0.0-15-generic)= f56839a4642eb97e06e5efb0bc74f4dc
me@GAMMA:~$ cat ~/3.0.0.15x86 
MD5(/boot/vmlinuz-3.0.0-15-generic)= f56839a4642eb97e06e5efb0bc74f4dc
me@GAMMA:~$ sudo openssl dgst -md5 /boot/vmlinuz-3.0.0-15-generic > ~/3.0.0.15x86

So the Linux kernel acts very much like the Mach Kernel in OSX in that it is a 32-bit executable that switch to 64-bit mode when needed. The question then is why am I getting an error about "Exec format error"?

This Post however seems to indicate that the runaway loop modprobe from the Kernel Panic indicates that this is indeed a 32/64 bit issue. Grub must have some way of telling the kernel the bitlength, maybe it is in an associated file in /boot, looking into that tomorrow.

quick update:
The 32-bit and 64-bit kernels hash differently, but file reports them both to be x86.

< MD5(/boot/vmlinuz-3.0.0-15-generic)= f56839a4642eb97e06e5efb0bc74f4dc
 ---
> MD5(/boot/vmlinuz-3.0.0-15-generic)= cee6cd7db9016ee8531be92504ac802b

So I need to determine how to install the kernel to somewhere other than /boot so that the two kernels don't clobber each other...