Recompiling a new kernel for the SheevaPlug

There are many reasons why you would want to use a custom kernel on the plug. As far as I am concerned, the lack of ext4 and ntfs support in the kernel provided with the otherwise excellent SheevaPlug installer 1.0 system make a clear case for a custom kernel. After all, you might wanna mount external HDDs that are NTFS formatted, or, more important, you might want to install/access a Linux system running with ext4 (which I hear is nicer on flash based media devices than ext3), as we will do when we install armedslack on an SDIO card.


Setting up a cross-compiler for the SheevaPlug on Linux x86_64

While it is definitely possible to recompile a kernel on the SheevaPlug itself, let us just consider the raw compilation times observed when compiling a 2.6.31.2 kernel (exact same .config) on the Sheeva and on a more up to date PC:
- Quad Core PC (Slackware 13.0 x86_64 w/ ext4 fs on a RAID5 array) - 3:03 mins (183 secs)
- SheevaPlug (Ubuntu 9.04 w/ ext4 fs on a high speed SDIO card) - 57:42 mins (3462 secs)

That's one full order of magnitude right there (or about 20x slower)! And if you need a further case for cross-compilation, remember that extracting the kernel source takes ALOT of space. You will run out of space if you try to extract it on the internal flash NAND - it's just too big for what's left of the 512 MB space, and even on external media, you might want to think twice about using the limited space you have there.

If you're familiar with this blog, you know that there's no way we'll want to use any of the old, likely outdated, tools that came with the SheevaPlug SDK CD or internet binaries got compiled by people we don't know (and therefore don't trust!). Besides, the Cross compiler tgz I got on the CD has a big fat CRC error to start with, and it's 32 bit while our cross compiling rig is running Linux x86_64. If what we've seen with the 32 bit openocd binaries is any indication of what's in store for 64 bit users, we might as well cut to the chase and ignore anything that's not 64 bit.
What we do want, and always will want, is the most cutting edge and up to date technology we can lay our hands on. Therefore, we will of course recompile our own cross compiler for the SheevaPlug using the latest gcc, glibc and binutils.

Now, we could of course dive in straight ahead, and recompile, by hand, the whole cross-compiling shebang, but let's face it: your time is precious, so is mine, and if you just dive in head first in trying to produce a cross compiler from the GNU vanilla tarballs, whilst having no previous experience in the matter, you'll find soon enough that:

1. Compiling a cross compiler manually is absolute hell!
2. glibc and gcc have to be the worst offenders for breaking things that worked perfectly fine before with newer versions. All those rumours you've heard about needing to apply a massive amount of patches to gcc and gblic before you can compile them as cross-compilers? They're true!

Fact of the matter is I wasted more than one afternoon yesterday on the "Heck, I'm smarter than that - I'll go manual!" approach, and, while I now have a clearer idea of how one might get a cross-compiler actually compiled from scratch, I'll just say this: Do yourself a favour - if you value your time, don't waste it in trying to create a cross-compiling toolchain by hand.
Now, if you really, really, insist on compiling a whole toolchain manually, then my advice is to try to stick as close as possible to this guide (trust me!), but don't come whining that I didn't warn you you were about to step through the gates of hell beforehand...


Using CrossTool-NG

The MUCH better approach, to automate the cross compilation setup process is to use Yann E. Morin's most excellent Crosstool-NG. And, yes, despite being an automated tool, it will not compromise on our using the latest versions of gcc, glibc, binutils, etc.

So off we go:

cd /usr/src
mkdir sheeva
cd sheeva
wget http://ymorin.is-a-geek.org/download/crosstool-ng/crosstool-ng-1.4.2.tar.bz2
tar -xjvf crosstool-ng-1.4.2.tar.bz2
cd crosstool-ng-1.4.2
./configure
make
make install
cd ..
ct-ng menuconfig

Well, isn't that nice - a kernel like configuration menu. Alrighty then, here are the options you want to change for the SheevaPlug:

• Paths and misc options
• Try features marked as EXPERIMENTAL: check
• Prefix directory: "/usr/local/${CT_TARGET}" (this will result in all the tools residing in /usr/local/arm-unknown-linux-gnueabi) or, preferred: "/usr/local/sheeva". One important note though: DO NOT USE /usr/local or any directory containing data you plan to keep. THIS DIRECTORY WILL BE ERASED. You have been warned.
  
• Use LAN mirror: Let's try to use a mirror there, so - check
• Prefer the mirror: check
• Base URL: ftp://ftp.heanet.ie/mirrors/ftp.gnu.org/gnu/ (this is an Irish mirror - not entirely sure it works, as the downloads didn't seem much faster, but worth a try)
  
• Stop after extracting tarballs: check (we'll need to patch glibc!)
• Number of parallel jobs: 2x the number of cores you have. In my case that would be 8 (quad core)
• Maximum log level to see: change from "INFO" to "EXTRA"
• Target Options
• Target Architecture: arm
• Use EABI: check
• Emit assembly for CPU: arm926ej-s
• Floating point: change from "hardware (FPU)" to "software"
  
• Toolchain options: don't change anything
• Operating System:
• Target OS: change from "bare-metal" to "linux"
• Get kernel headers from: change from "kernel's 'headers_install'" to "Use custom headers"
• Path to custom headers directory/tarball: "/usr/src/sheeva"
• GMP and MPFR:
• GMP and MPFR: check
• GMP version: 4.2.4 (latest)
  
• MPFR version: 2.4.1 (latest)
• binutils
• binutils version: 2.19.51.0.2
• C compiler:
• gcc version: 4.3.3
• C++: check
• C-library:
• C library: glibc
• glibc version: 2.9
• Threading implementation to use: make sure "ntpl" is set.
  

The rest should be left untouched. For your reference, I am providing the Crosstool-NG .config file I used. Please note that if you use a recent version of glibc, you MUST have a threading implementation set, but you can NOT use linuxthreads, as it's been deprecated (you WILL get errors if you do select it), so the only option to use in nptl.


Kernel headers & glibc 2.9 patch

Before you can run the build, we still need to setup our kernel headers.
At the time of this writing, the most recent kernel is linux-2.6.31.2, so just download and extract it somewhere. Then, within the linux-2.6.31.2/ directory, run the command:

make headers_install ARCH=arm INSTALL_HDR_PATH=/usr/src/sheeva

The INSTALL_HDR_PATH should match exactly what you used for "Path to custom headers directory/tarball" in the menuconfig of Crosstool-NG.
You should now find that you have an "include" directory containing the relevant kernel headers in /usr/src/sheeva

Once you have everything setup, you need to go back to the sheeva directory (because this is where the ct-ng config resides) and launch the build with:

cd /usr/src/sheeva
ct-ng build

If you followed what we did above, you will note that we checked the "Stop after extracting tarballs" option, and indeed the build process stops after "Extracting and patching toolchain components". The reason we did that is because glibc 2.9 needs to be manually patched to prevent the following error (undefined reference to `_begin') from occurring:

[ALL  ]    /usr/src/sheeva/targets/arm-unknown-linux-gnueabi/build/build-libc/elf/librtld.os: In function `_dl_start_final':
[ALL  ]    raise.c:(.text+0x54c): undefined reference to `_begin'
[ALL  ]    /usr/src/sheeva/targets/arm-unknown-linux-gnueabi/build/gcc-core-shared/lib/gcc/arm-unknown-linux-gnueabi/4.3.3/../../../../arm-unknown-linux-gnueabi/bin/ld: /usr/src/sheeva/targets/arm-unknown-linux-gnueabi/build/build-libc/elf/ld.so: hidden symbol `_begin' isn't defined
[ALL  ]    /usr/src/sheeva/targets/arm-unknown-linux-gnueabi/build/gcc-core-shared/lib/gcc/arm-unknown-linux-gnueabi/4.3.3/../../../../arm-unknown-linux-gnueabi/bin/ld: final link failed: Nonrepresentable section on output
[ALL  ]    collect2: ld returned 1 exit status
(...)
[ERROR]    Build failed in step 'Installing C library'

This known problem is documented here and if you have a look at the diff file, you'll see that we can easily manually patch this problem by editing "targets/src/glibc-cvs-2.9/elf/Makefile" and replace, on line 314:

-e 's/\. = 0 + SIZEOF_HEADERS;/& _begin = . - SIZEOF_HEADERS;/' \

with

-e 's/\. = .* + SIZEOF_HEADERS;/& _begin = . - SIZEOF_HEADERS;/' \

Building the cross-compiler

Once you have made this change, just launch ct-ng menuconfig again (from the /usr/src/sheeva directory, as this is where the config file resides) and remove the checkmark on "Stop after extracting tarballs" (in the "Paths and misc options" menu)

Now, you can launch the actual build of the tools with

ct-ng build

This will take a little while (about 15 mins on my machine) but should complete successfully. If you have a multiple core system, hopefully you didn't forget to change the "Number of parallel jobs" option (which is equivalent to the -j option of make, i.e. number of concurrent jobs to run at once), as this dramatically reduces the time it takes to compile the toolchain. The -j option should always be your friend when compiling a large source, like a Linux kernel for instance.

Time for a tea or coffee break!

Note that, near the end of the compilation, you might get the error "[ERROR] libtool.m4: error: problem compiling FC test program". From what I gather however, FC is a test for the Fortran 90 compiler, which we couldn't care less about, so, unless you got another error, I will consider this build process of success, and you should now have the whole set of cross compiler tools in /usr/local/sheeva/bin. Time to compile ourselves a new kernel!


Installing U-boot mkimage

While we now have the tools to create arm binaries, we're still missing the last piece of the puzzle to boot a kernel, and that is the mkimage utility (which is called at the end of a successful kernel compilation), which turns a newly crafted kernel into something U-Boot can actually launch.

You guessed it, no way in hell we're gonna use the Marvell provided mkimage tool - we're gonna recompile our own from source of course!
And once more, the good guys at Debian (never underestimate a Debianer!) provide us with what we need:

cd /usr/src/sheeva
wget http://ftp.de.debian.org/debian/pool/main/u/uboot-mkimage/uboot-mkimage_0.4.tar.gz
tar -xzvf uboot-mkimage_0.4.tar.gz
make
make install

Easy as a pie!


Cross-Compiling a new kernel for the SheevaPlug

At long last, we're ready to compile us some custom kernel!
At this stage, we can more or less follow the Compiling Linux Kernel for the Plug Computer tutorial from openplug.org . Henceforth:

cd /usr/src/sheeva/linux-2.6.31.2
make ARCH=arm kirkwood_defconfig
make ARCH=arm menuconfig

There, change the options that you want. If you plan to keep using UBIFS (which you should have switched to during the SheevaPlug Installer 1.0 upgrade), then make sure that you enable BOTH:

• Device Drivers ---> Memory Technology Device (MTD) support ---> UBI - Unsorted block images ---> Enable UBI
  
• File systems ---> Miscellaneous filesystems ---> UBIFS file system support
• Device Drivers ---> Generic Driver Options, and make sure that both "Create a kernel maintained /dev tmpfs (EXPERIMENTAL)" and "Automount devtmpfs at /dev" are enabled. Without those, you won't see init messages on the serial console after the root filesystem is mounted.

With this set, and the other default options from kirkwood_defconfig, you should be able to produce a usable kernel. Now is obviously a good time to add ext4 & NTFS suppot, and remove features (802.11 stack) that you do not want. Finally, we're ready to compile, and the kernel makes cross compilation very easy for us, as its CROSS_COMPILE feature enables us to specify exactly which tools it should use instead of the default ones. Thus, because we have an arm-unknown-linux-gnueabi-gcc, arm-unknown-linux-gnueabi-ld, etc. in /usr/local/sheeva/bin, we will use the line:

make -j8 ARCH=arm CROSS_COMPILE=/usr/local/sheeva/bin/arm-unknown-linux-gnueabi- uImage

and in about 3 minutes (vs. one hour if done on the SheevaPlug) you should end up with something like

Image Name:   Linux-2.6.31.2
Created:      Tue Oct  6 14:50:51 2009
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    3151992 Bytes = 3078.12 kB = 3.01 MB
Load Address: 0x00008000
Entry Point:  0x00008000
Image arch/arm/boot/uImage is ready

Building the modules

You probably still have a few modules to compile as well, which you can do as follows

make -j8 ARCH=arm CROSS_COMPILE=/usr/local/sheeva/bin/arm-unknown-linux-gnueabi- modules
make -j8 ARCH=arm CROSS_COMPILE=/usr/local/sheeva/bin/arm-unknown-linux-gnueabi- INSTALL_MOD_PATH=./modules modules_install

Just don't forget that you'll need to copy the content of the ./modules directory to your Ubuntu image to keep the kernel happy


Testing the new kernel on the plug

In a next post, we'll see how to actually flash that kernel on the plug. Before we do that though, we want to ensure that it can boot our current system, so we'll try to launch it from an USB stick.
Off we go then to copy your shiny new kernel to an USB stick with

mount /dev/sde1 /mnt/usb
cp arch/arm/boot/uImage /mnt/usb/
sync
umount /mnt/usb

For the test part, we'll need the serial console ("screen /dev/ttyUSB0 115200,-crtscts" or similar), with which we first reboot our system, and interrupt the autoboot to get to the Marvel >> prompt. From there, the following set of commands should boot the existing system with our new kernel:

usb start
fatload usb 0 0x00800000 uImage
bootm 0x00800000

Success? Then my work here is done.