>NIL:'s Techno Feats

Help, my RAID array does not complete synchronization!

2012-01-10T13:54:00.000Z

Let us suppose the following situation: You have a Linux server with a software RAID1 array (md) and, for one reason or another (mostly because your are a lazy admin, admit it!), both disks are reporting unreadable sectors, either through SMART or through actual failed readout attempts.

So you installed a 3rd good disk, set it as a spare, then failed one of the 2 bad ones to initiate synchronisation onto the good new disk. However, all hell breaks lose as you find out your synchronisation doesn't complete (/proc/mdstat reports U_ or _U) and instead of ignoring the unreadable sectors as it should, md decides that it cannot continue.

Worse, if you look at your dmesg, you find out that it is being polluted by a continuous stream of:

RAID1 conf printout:
--- wd:1 rd:2
disk 0, wo:0, o:1, dev:sda1
disk 1, wo:1, o:1, dev:sdb1

Help!!!!

OK, first of all, since this information is quite hard to find, especially if you are in a hurry, here are what the abbreviations above mean:

wd: working disks
rd: raid disks
wo: write-only (if set to 1, this usually indicates a problem, and that data duplication doe not occur for this device)
o: online

Obviously, wd:1 as well as wo:1 for the second disk is not something we want to see. Why can't our good spare disk be added as R/W to the gorram array? Heck, if the problematic disk fails, that single-handedly contains our up-to-date data now, we will be in big trouble. What's the point of providing redundancy, really, if md fails to synchronize as soon as there's one measly sector it cannot read!

It's a bird! It's a plane! No, it's hdparm!

Well, the sad truth of md on Linux (which may have improved with newer versions) is that it isn't resilient at all when it comes to unreadable sectors during sync. I guess the developers decided that, since the point of redundancy is to always have at least one good set of data, they didn't need to focus on situation where the "good" set of data may also have some corruption, and therefore never planned for anything but try and re-read an unreadable sector forever, until the disk magically repairs itself (right... fat chance!).

Now (and for the rest of this post I will mostly be following the excellent information provided by Bas on his blog) to compensate for that oversight, the trick is to have md read the problematic sectors one way or another, so that the synchronisation can complete. May sound easier said than done but most of the time it shouldn't be an issue, as recent disks with SMART are engineered with a set of spare sectors, to be allocated in replacement of unreadable or unwritable ones for exactly this kind of situation. The issue however is that reallocation of sectors only occurs on write access.

What this means then is that, while the disk has the technology to "fix" itself, as long as you are only attempting to read the problematic sectors, reallocation will not be triggered and you will continue to get read errors. Thus, you must manually issue a write to the problematic sector(s) to trigger the "recovery" mechanism (NB: I'm using "fix" and "recovery" loosely, as you can of course not recover data from these sectors if they are reallocated, therefore will end up with some corrupted data).

This can be confirmed by checking the Offline_Uncorrectable (#198) and Reallocated_Sector_Ct (#5) reports from SMART:

# smartctl -A /dev/sda
smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   051    Pre-fail  Always       -       105
  2 Throughput_Performance  0x0026   054   054   000    Old_age   Always       -       2759
  3 Spin_Up_Time            0x0023   084   084   025    Pre-fail  Always       -       4989
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       10
  5 Reallocated_Sector_Ct   0x0033   252   252   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   252   252   051    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0024   252   252   015    Old_age   Offline      -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       11496
 10 Spin_Retry_Count        0x0032   252   252   051    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   252   252   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       10
191 G-Sense_Error_Rate      0x0022   252   252   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0022   252   252   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0002   064   060   000    Old_age   Always       -       32 (Lifetime Min/Max 20/40)
195 Hardware_ECC_Recovered  0x003a   100   100   000    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   252   252   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   252   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   252   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0036   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x002a   100   100   000    Old_age   Always       -       2
223 Load_Retry_Count        0x0032   252   252   000    Old_age   Always       -       0
225 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       10

If you see a zero at the end of these attributes but the disk still reports that it has trouble reading sectors, it indicates that the sector reallocation process hasn't kicked in yet, and needs to be triggered manually.

The first order of the day then is to find the address of the sector(s) we should trigger a write to. This is fairly easy, as all you need to do is run a SMART test, with something like smartctl -t long /dev/sda and write down the first sector address where a read error is reported:

# smartctl -a /dev/sda
(...)
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       60%     10864         293039329
(...)

Once we have that address, we could of course use dd, but an even simpler approach is to use a recent version of hdparm, as it adds easy support for reading/writing a single sector.

First thing to try with hdparm then, is confirm that we have a problem accessing that sector:

# hdparm --read-sector 293039329 /dev/sda

/dev/sda: Input/Output error

This confirms what the SMART test reported. You can try a few more read attempts, to validate that the sector is busted, and then, you can issue a write so that the disk finally realizes it should reallocate that sector. Note that, because the operation obviously means destroying existing data, hdparm requires you to add a --yes-i-know-what-i-am-doing flag to issue the write, hence:

# hdparm --yes-i-know-what-i-am-doing --write-sector 293039329 /dev/sda

/dev/sda: re-writing sector 293039329: succeeded

You can then issue a read again, which will confirm that the sector has been reallocated:

# hdparm --read-sector 293039329 /dev/sda

/dev/sda:
reading sector 293039329: succeeded
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000

If you issue smartctl -A again, you should also see that the sector has been reallocated:

198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -        1

It's usually a good idea to use hdparm to read adjacent sectors as well, and correct them as needed, then repeat the operations above until the SMART self test completes without error and you have smoked out all the problematic sectors. At this stage, if you issue a resync of the array with the new disk, it should complete successfully and redundancy will be restored. Time to order another replacement and check your data for corruption. But at least, you are redundant again.

Addons

To get details of your md array, you can use mdadm --detail. Eg.

# mdadm --detail /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Tue May  6 18:43:16 2008
     Raid Level : raid1
     Array Size : 130030016 (124.01 GiB 133.15 GB)
  Used Dev Size : 130030016 (124.01 GiB 133.15 GB)
   Raid Devices : 2
  Total Devices : 3
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Tue Jan 10 13:42:29 2012
          State : clean
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

           UUID : 0be47c81:ede086ae:0c460403:d81de298
         Events : 0.3658859

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3

       2       8       35        -      spare   /dev/sdc3

You are strongly encouraged to check your syslog or messages for reports of I/O issues, especially if you want to locate the data that may have been affected.
This method is not guaranteed to work! Sometimes a SMART test will report a read error but a readout of the sector using hdparam will work fine, so you won't be able to get the disk to reallocate it. However, tis shouldn't matter too much for md resync which is what we are interested in here.
If your disk has a lot of unreadable sectors, it is possible that you may run out of spare sectors for reallocation. It's hard to say how many spare sectors are made available by hard drive manufacturers, but I assume it isn't that many.

You may have a problem recompiling a recent version of hdparm on some older Linux systems:

fallocate.c: In function ‘do_fallocate_syscall’:
fallocate.c:39: error: ‘__NR_fallocate’ undeclared (first use in this function)
fallocate.c:39: error: (Each undeclared identifier is reported only once
fallocate.c:39: error: for each function it appears in.)
make: *** [fallocate.o] Error 1

If that is the case, just add:

#define __NR_fallocate 285

in fallocate.c

Some disks seem to be smart enough (no pun intended) to do further correction, once they have registered Offline_Uncorrectable sectors, so you may actually find out that, after a few hours, the value of Offline_Uncorrectable falls back to zero, and still the sectors can be read or written with extended SMART tests not reporting any issue. Pretty neat, but I still wouldn't entirely trust the disk...

Using LILO to boot disks by UUID

2012-01-04T12:07:00.002Z

If you're plugging USB drives in an out and using LILO to boot a Linux distro (eg. Slackware) you may have ended up with a kernel panic because your /dev/sd# were shuffled around and the kernel was no longer able to find its root partition on the expected device. Of course, having Linux failing to boot just because you happened to plug an extra drive sucks big time, so we want to fix that.

The well known solution of course it to use UUIDs or labels, since these are fixed. However, while recent versions of LILO are supposed to support root partitions that are identified by UUID/Label, in practice, this doesn't work UNLESS you are using an initrd disk. I'm not sure who of LILO or the kernel is responsible for this new layer of "suck" (I'd assume the kernel, since the expectation is that LILO is using the dev mappings that are being fed by the kernel), but I can only say that there really are some areas of Linux that could still benefit from long awaited improvements...

Thus, to be able to use UUIDs or labels for your root partition in LILO, you must boot using an initrd. Worse, as previously documented, you will most likely need to compile a new kernel that embeds the initrd, lest you want to run into the following issue while running LILO:

Warning: The initial RAM disk is too big to fit between
the kernel and the 15M-16M memory hole.

In practice (as also illustrated by this post), this means you will need to:

Create an initrd cpio image that can be embedded into a kernel with:

cd /boot
mkinitrd -c
cd initrd-tree
find . | cpio -H newc -o > ../initrd.cpio

Recompile a kernel, while making sure that you have the General Setup → Initial RAM filesystem and RAM disk (initramfd/initrd) support selected, and then set General Setup → Initramfs source file(s) to /boot/initrd.cpio.
Edit your /etc/lilo.conf and add an append = "root=UUID=<YOUR-DISK-GUID>" to your Linux boot entry. An example of a working lilo.conf is provided below. Note that you probably also want to use a fixed IDs for boot=, so that running LILO is also not dependent on the current /dev/sd# organization..
Run LILO, plug drives around and watch in amazement as your system still boots the Linux partition regardless of how the drives are assigned

Example lilo.conf:

# Start LILO global section
boot = /dev/disk/by-id/ata-ST3320620AS_ABCD1234
compact
lba32
# LILO doesn't like same volume IDs of RAID 1
disk = /dev/sdb inaccessible
default = Windows
bitmap = /boot/slack.bmp
bmp-colors = 255,0,255,0,255,0
bmp-table = 60,6,1,16
bmp-timer = 65,27,0,255
# Append any additional kernel parameters:
append=" vt.default_utf8=1"
prompt
timeout = 35
# End LILO global section

image = /boot/vmlinuz
  append = "root=UUID=2cc11aaf-f838-4474-9d9a-f3881569f97c"
  label = Linux
  read-only
image = /boot/vmlinuz.rescue
  append = "root=UUID=2cc11aaf-f838-4474-9d9a-f3881569f97c"
  label = Rescue
  read-only
other = /dev/sda
  # Windows doesn't go to S3 sleep and has issues with backup,
  # unless it sees its disk as first in BIOS...
  boot-as = 0x80
  label = Windows
other = /dev/disk/by-id/ata-ST3320620AS_ABCD1234-part4
  label = OSX

Enabling serial console on Linux Slackware

2011-08-31T12:35:00.013+01:00

I'm doing this frequently enough to warrant a post.

Make sure you use a kernel with console enabled

Confirm that your serial tty's are detected with dmesg | grep tty:

[    0.000000] console [tty0] enabled
[    1.323949] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    1.568561] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    1.592267] 00:0c: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    1.614293] 00:10: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A

Edit /etc/securetty and uncomment the lines matching the ttyS you got from dmesg. Else you won't be able to logon as root.

Edit your /etc/inittab and uncomment the lines:
```
s1:12345:respawn:/sbin/agetty -L ttyS0 9600 vt100
s2:12345:respawn:/sbin/agetty -L ttyS1 9600 vt100
```
While you're at it, you probably want to change the 9600 vt100 to 115200 linux, if you use putty in serial mode to connect for instance.

If you want the boot messages on serial as well (recommended!), make sure you append a console= line to your kernel. For instance, if using LILO and for 115200 bauds, you would add a line:
```
append = "console=ttyS0,115200"
```

Extracting a single DTS/AC3 channel from an MKV file as a PCM WAV

2011-08-03T13:09:00.002+01:00

Well, since audio and video software editors are still not there yet, a quick recepy.

Say you have a H264/DTS MKV video file and you want to extract the center channel, as WAV, for easy editing.

Download tsMuxeR and extract the audio stream as a .dts (eg. "multichannel.dts"), using the GUI
Download eac3to and run the following command to convert to a multichannel PCM WAV: eac3to.exe multichannel.dts multichannel.wav
Download wavosaur and open the multichannel WAV. Then kill the channels you don't need, edit the file, etc.

None of the programs above use an installer - they can be extracted and run directly.

Slackware 13.37 and minicom

2011-06-30T17:04:00.002+01:00

Doesn't work by default. Wanna know why? /etc/minirc.dfl is missing a Line Feed. Yes, really, all you need to do is add an extra blank like there.

Installing OSX (Snow Leopard) + Linux (Slackware 13.37 x86_64) on a GUID/GPT disk, with Software RAID enabled (ICH8R) - Part 5

2011-05-25T16:34:00.012+01:00

Focusing on Linux + LILO setup, to (hopefully) conclude this series.

At this stage, you are supposed to have a standalone GPT/GUID disk, with a bootable OSX (Chameleon, Chimera), as well as free space for a Linux installation.

More disk partitioning

First order of the day is to pick up the latest Slackware distro (13.37 at the time of this post) and fire it up. Earlier versions of Slackware cannot handle GPT disks, and I don't believe they include gdisk either, so make sure you pick the very latest.

Now, since we seeded the partitioning from OSX and left some free space, when firing up gdisk (using sdc here as sda+sdb are used by Windows 7 in ICH8R RAID1) you'll be greeted with the following:

# gdisk /dev/sdc
GPT fdisk (gdisk) version 0.6.14

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/sdc: 234441648 sectors, 111.8 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 592CD063-7F0A-4F52-81EC-A58C5D3F859C
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 234441614
Partitions will be aligned on 8-sector boundaries
Total free space is 175418693 sectors (83.6 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              40          409639   200.0 MiB   EF00  EFI System Partition
   2          409640        59022927   27.9 GiB    AF00  Snow Leopard

Command (? for help): n
Partition number (3-128, default 3): 
First sector (34-234441614, default = 59022928) or {+-}size{KMGTP}: 
Last sector (59022928-234441614, default = 59022928) or {+-}size{KMGTP}: +30G
Current type is 'Linux/Windows data'
Hex code or GUID (L to show codes, Enter = 0700):
Changed type of partition to 'Linux/Windows data'

Command (? for help): p
Disk /dev/sdc: 234441648 sectors, 111.8 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 592CD063-7F0A-4F52-81EC-A58C5D3F859C
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 234441614
Partitions will be aligned on 8-sector boundaries
Total free space is 112504133 sectors (53.6 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              40          409639   200.0 MiB   EF00  EFI System Partition
   2          409640        59022927   27.9 GiB    AF00  Snow Leopard
   3        59022928       121937487   30.0 GiB    0700  Linux/Windows data

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!!

Do you want to proceed, possibly destroying your data? (Y/N): y
OK; writing new GUID partition table (GPT).
The operation has completed successfully.

Note 1: You'll notice that there's a 200 MB EFI System Partition. If you want OSX to be happy, even if the OSX bootloader shouldn't need it, better leave it alone.

Note 2: You'll have noticed that when firing gdisk, it mentioned something about a "protective MBR". This is because GPT and MBR can coexist on the same disk, and therefore an MBR-only aware utility could potentially destroy data. We'll put the MBR+GPT coexistence to good use later on in this post.

Note 3: If you leave OSX's disk utility create the partition for the Linux system and try to change the type in gdisk, the installer might not see your Linux system partition as a valid target. If that occurs, you should delete the non OSX/EFI System partitions in gdisk (d command), and recreate them.

Slackware and LILO setup

The Slackware install is as straightforward as usual, so I'm not going to comment on it. At the end, just install LILO on the MBR of your standalone OSX/Slackware disk (here /dev/sdc) as you'd normally do. Reboot, and you oughta be able to access your newly installed Linux system.

So how about we add OSX boot to our etc/lilo.conf and be done with this whole exercise then? The OSX bootloader (Chameleon, etc.) should be able to take the relay after LILO handoff. And while we're at it, we'll also add Windows boot from /dev/sda.
Off we go then and add the following at the end of /etc/lilo.conf:

other = /dev/sda
  label = Windows
other = /dev/sdc2
  label = OSX

And now the fun begins...

LILO troubles

# lilo

Reference:  disk "/dev/sdb"  (8,16)  0810

LILO wants to assign a new Volume ID to this disk drive.  However, changing
the Volume ID of a Windows NT, 2000, or XP boot disk is a fatal Windows error.
This caution does not apply to Windows 95 or 98, or to NT data disks.

Is the above disk an NT boot disk? [Y/n]

Hell to the no! What on earth is going on here? Well, a little googling around tells you that LILO is unhappy because our mirrored disks (/dev/sda and /dev/sdb) bear the same volume ID. Two solutions there:

Solution 1: add a section:
```
disk = /dev/sdb
  inaccessible
```
to lilo.conf to make it ignore the second mirrored disk altogether. In this case LILO will issue the warning Warning: bypassing VolumeID scan of drive flagged INACCESSIBLE: /dev/sdb but proceed.

Solution 2 (better): enable dmraid access to your drive. The thing is that the Slackware installer actually saw our mirrored volume alright (it listed our device automatically as /dev/md126*) but when booting our newly installed Slackware, the dm volume was no longer there. In case you wonder, this is because the Slackware installer rc.S script issues a /sbin/mdadm -A -s after fuse has been launched, which enables autodetection of mirrored drives.
Therefore, if you just run that same command, the RAID 1 array will be detected and LILO won't complain about Volume IDs.
```
# /sbin/mdadm -A -s
mdadm: Container /dev/md/imsm0 has been assembled with 2 drives
mdadm: Started /dev/md/WindowsRAID1_0 with 2 devices
```
Probably a good idea to add that line in your system's rc.S script as well.

Now, with the Windows RAID issue being sorted, let's try again, shall we:

# lilo
Added Linux *
Added Windows
Warning: Device 0x0820: Inconsistent partition table, 2nd entry
  CHS address in PT:  0:0:0  -->  LBA (-1)
  LBA address in PT:  0  -->  CHS (0:0:1)
Fatal: Either FIX-TABLE or IGNORE-TABLE must be specified
If not sure, first try IGNORE-TABLE (-P ignore)

Son of a "£$%^&!! What's going on there?

Booting GPT partitions using MBR tools

I'll cut to the chase, our actual problem here is that LILO (as well as GRUB) is an MBR bootloader, not a GPT one. Therefore it actually uses the MBR partition table, and if you look at this table, you'll see that as far as MBR is concerned, /dev/sdc2 is nowehere to be found:

# fdisk -l /dev/sdc

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdc: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x41b82465

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1   234441647   117220823+  ee  GPT

Wait a minute, if that's the case, how comes LILO was able to boot our Linux GPT partition then? Isn't that one GPT too?
Aha! Well, the trick here is that, when booting the system, LILO doesn't need to know how the partition or filesystem is setup. Since the kernel file is always available to the Linux system you want to run LILO from (duh!), LILO can point to the actual kernel image sectors directly, so it actually bypasses the whole partition table (as well the filesystem), and just provides a list of raw blocks to read. Haven't you learned something interesting today? And that is why even a non GPT aware bootloader like LILO is able to boot from a kernel residing on a GPT Linux system partition. But of course, as soon as you leave the system partition, then LILO needs an MBR reference, to know where the start of the target partition is located, which it cannot find.

gptsync to the rescue!

At first glance it looks like we may have to abandon LILO. As far as I know, GRUB is unlikely to perform any better though (it's also an MBR based bootloader), but there exist projects targeted at GPT such as eLILO, though these may also require an EFI platform as well, which may also render them useless for our usage scenario.

Now, remember what I said about protective MBR tables for GPT disks and how both MBR and GPT could coexist? How about we created an MBR table that contains the same partition information as our GPT one then? Well, that's exactly what the rEFIt's gptsync tool is all about. This is a tool that can take a GPT partition table and create an exact MBR clone of it, so that MBR aware tools get the relevant information about the partitions.
Of course, this can only work if the GPT partition doesn't deviate too much from MBR compatibility requirements, which in most case (if your HDD is less than 2 TB as this is the hard limit for maximum MBR), shouldn't be a problem.
At this stage you have two options: pick up the binary gptsync executable for OSX from the Mac disk image or recompile your own from source in Linux. If you want a precompiled gptsync for OSX you can find one here (direct mirror), but considering that we're in Linux right now, we'll go the recompilation route. After having downloaded and extracted the latest refit package, just navigate to the gptsync directory and run make -f Makefile.unix. As these utilities are fairly simple, this should produce a gptsync and showpart executable in the same directory. Might be a good idea to see what showpart reports before going further, right?

# ./showpart /dev/sdc

Current GPT partition table:
 Warning: Unknown GPT spec revision 0x00010000
Floating point exception

Dammit! Does no single tool work as expected out of the box these days, or is it just that I'm always one step ahead in trying cutting edge stuff? Long story short, you get this error on 64 bit Linux environments because the tools use unsigned long for 32 bit unsigned, and unsigned long is not 32 bit on x86_64. Easiest way I found to fix this is to include <stdint.h> in gptsync.h and then change the UINT8, UINT16, UINT32, UINT64 typedefs to use uint8_t, uint16_t, etc.... Once this is done:

# ./showpart /dev/sdc

Current GPT partition table:
 #      Start LBA      End LBA  Type
 1             40       409639  EFI System (FAT)
 2         409640     59022927  Mac OS X HFS+
 3       59022928    121937487  Basic Data

Current MBR partition table:
 # A    Start LBA      End LBA  Type
 1              1    234441647  ee  EFI Protective

MBR contents:
 Boot Code: LILO

Partition at LBA 40:
 Boot Code: None (Non-system disk message)
 File System: FAT32
 Listed in GPT as partition 1, type EFI System (FAT)

Partition at LBA 409640:
 Boot Code: None
 File System: Unknown
 Listed in GPT as partition 2, type Mac OS X HFS+

Partition at LBA 59022928:
 Boot Code: None
 File System: ext4
 Listed in GPT as partition 3, type Basic Data

Much better! Off we go then:

# ./gptsync /dev/sdc

Current GPT partition table:
 #      Start LBA      End LBA  Type
 1             40       409639  EFI System (FAT)
 2         409640     59022927  Mac OS X HFS+
 3       59022928    121937487  Basic Data

Current MBR partition table:
 # A    Start LBA      End LBA  Type
 1              1    234441647  ee  EFI Protective

Status: MBR table must be updated.

Proposed new MBR partition table:
 # A    Start LBA      End LBA  Type
 1              1       409639  ee  EFI Protective
 2         409640     59022927  af  Mac OS X HFS+
 3 *     59022928    121937487  83  Linux

May I update the MBR as printed above? [y/N] y
Yes

Writing new MBR...
MBR updated successfully!

Final LILO tuning

Alrighty. Time for a new run at LILO:

# lilo
Added Linux *
Added Windows
Warning: Device 0x0820: Inconsistent partition table, 2nd entry
  CHS address in PT:  1023:254:63  -->  LBA (16450559)
  LBA address in PT:  409640  -->  CHS (25:127:15)
Fatal: Either FIX-TABLE or IGNORE-TABLE must be specified
If not sure, first try IGNORE-TABLE (-P ignore)

Well, actually this is not entirely unexpected considering that GPT does away with the MBR requirements of having partitions starting and ending on a cylinder boundary, so of course, if you don't fine tune your GPT partition, you're likely to have an MBR partition table that doesn't meet the MBR specs. However, in this case, we can safely ignore that issue, by making sure we have the following at the top of our /etc/lilo.conf

lba32        # LBA addressing should be default anyway
ignore-table # ignore CHS vs LBA conflicts in the MBR

This time:

# lilo
Added Linux *
Added Windows
Warning: Device 0x0820: Inconsistent partition table, 2nd entry
  CHS address in PT:  1023:254:63  -->  LBA (16450559)
  LBA address in PT:  409640  -->  CHS (25:127:15)
Warning: The partition table is *NOT* being adjusted.
Added OSX
2 warnings were issued.

Yay, at long last, our bootloader installed!
NB: if you get an "invalid block" error on your OSX partition in LILO, this means that your OSX bootloader (Chameleon, Chimera) has not been properly installed, so you will need to fix that first.

With LILO in place, you should find that you now have multiboot between Linux, Windows and OSX, and you didn't have to forfeit your ICH8R Windows partition in the slightest.

As is obvious from the length of these posts, this whole process took some fairly heavy lifting, but I hope that was worth it!

Installing OSX (Snow Leopard) + Linux (Slackware 13.37 x86_64) on a GUID/GPT disk, with Software RAID enabled (ICH8R) - Part 4

2011-05-25T16:29:00.002+01:00

Tuning the OSX installation

Just a placeholder for now, as I still need for figure a few things. This will encompass removing the need to update AHCI for RAID everytime the system performs an update, making OSX bootable on its own, installing additiona drivers, etc.

All of the above doesn't really need to be performed before the Linux installation, but it looks tidier to keep all the OSX related posts in the same breadth... Stay tuned.

Installing OSX (Snow Leopard) + Linux (Slackware 13.37 x86_64) on a GUID/GPT disk, with Software RAID enabled (ICH8R) - Part 3

2011-05-24T21:59:00.010+01:00

Alright, on to the actual OSX installation. This assumes that you have sorted out the matter of getting your SATA controllers detetected, and have booted your USB installation media with iBoot

Disk partitioning

From there on, this is the regular OSX installation process. After selecting your language and graciously ignoring the EULA, you should be greeted by a list of all the disks that the installation process is able to handle.
Unless you did something wrong in the previous process, you should have all your disks (though some of those may not appear until you run the disk utility - see next step). If you can't see them here at all, then you either screwed up your kext modification or your kext cache rebuilding. Shame on you!

In our case, we're going to use a new drive for the OSX + Linux installation, so the first order of business is to partition our HDD in GPT/GUID mode, using the OSX disk utility, which we prefer over using Linux's gdisk as OSX seems to have some installation quirks that gdisk alone doesn't seem to be able to address. I found out that if you use gdisk to modify the GPT partition, you're likely to have the OSX installer complaining that it cannot boot from it.

To parition the disk, simply launch "Disk Utility" from the Utilities menu at the top. Then select the destination disk and click the "Partition" button.

Now if anyone knows how one can make the OSX Disk Utility to accept user specified sizes, I'd like to hear from you as any attempt I tried to create a 40 GB single partition for OSX, while leaving the rest of the disk unused have failed. Gotta wonder what the point of providing a "Size" field is when it is blatantly ignored. The trick I used then was to go through the various "n-partitions" options, and pick the one that creates a first partition (for OSX) that is as close as possible to the size I actually want. Then just go through all the extra paritions and change their Format to "Free Space".

The points you want to pay attention to are:

Label your OSX partition with something descriptive - don't keep "Untitled 1" (though you can change that later)

Select "Mac OS Extended (Case-sensitive, Journaled)" for the Format (it's an UNIX based OS, it IS case sensitive dammit!). Now, if you ever plan to modify your OS files from Linux, then you may want to select a non Journaled filesystem, as the current HFS driver from Linux does not offer write access on Journaled.

MAKE SURE "GUID Partition Table" (a.k.a. GPT) is used by clicking the "Options" button. Though you should be able to get away with MBR, the whole point of this exercise is to have a GPT disk to play with, so that's what we're gonna use.

Once your partitioning is set double check that you are partitioning the right disk, click Apply and quit the Disk Utility program. The newly created partition should now appear as a valid destination target for the installation process, so just select it and click "Install" (or click Customize then install, as it may be a good idea to remove additional languages or printer support if you don't have use for it). You can now go grab a cup of your favorite beverage and come back when the installation process has completed.

Dude, where's my boot?

So the install process completed successfully, the machine rebooted (more or less as the reboot process can be a bit flaky), and iBoot sees the new OSX installation alright. "VICT..., oh wait, this doesn't boot at ALL! AGHHH!!!".

Of course it doesn't you silly! To reiterate what I pointed out at the end of the previous post, the OSX installation process installs a clean, unmodified set of kexts, therefore all our good work about supporting the RAID controller on the USB key has been ignored. For the time being, we'll just fix that as we previously did, but instead of accessing the USB key, we'll do it on the OSX target drive:

Boot once more into your USB installation media with iBoot

go to Utilities→Terminal

issue the command "mount". It should list your OSX partition, indicate that it has write acces, and tell you where it is mounted (should be in /Volumes/Something)

cd /Volumes/<your_osx_partition>/System/Library/Extensions/AppleAHCIPort/Contents

Issue nano Info.plist (you do have nano on your installation key, right? If not, see the first post in this series).

Apply the same change as we did in previous post, namely change the <string>0x01060100&0xffffff00</string> part to <string>0x01060100&0xffffff00 0x01040000&0xffff0000</string> and save the file.

Now, as previously, we need to rebuild the kext cache. The only difference is that unlike the installation media, the cache for Snow Leopard is located at System/Library/Caches/com.apple.kext.caches/Startup, so the command you want to issue is:
```
kextcache -v 1 -t -l -a i386 -a x86_64 -m /Volumes/<your_osx_partition>/System/Library/Caches/com.apple.kext.caches/Startup/Extensions.mkext /Volumes/<your_osx_partition>/System/Library/Extensions
```

Quit the installer and restart the machine

Remove the USB key and boot into iBoot. Then select your OSX partition. This time it should boot alright and finalize the installation process. Wilkommen, Bienvenue, Welcome to cabaret indeed! But keep that USB installation media handy, as you are likely to need it again very soon.

Installing OSX (Snow Leopard) + Linux (Slackware 13.37 x86_64) on a GUID/GPT disk, with Software RAID enabled (ICH8R) - Part 2

2011-05-24T11:41:00.017+01:00

With both the iBoot CD and the OSX installation USB key rearing to go (see previous post), it's time to insert them both, and reboot, while making sure the CD/DVD drive is selected as your boot device. After a while, the iBoot boot selection screen should pop up, and provide you with the option to select "Mac OSX Install DVD" to boot from (which is actually our USB key) to start the installation process. NB: the name may be truncated, but if this is your first OSX install, just make sure you pick up the icon wit the Apple logo, as there should be only one.

Now, if your SATA controller was set to AHCI, or if you were using a JMicron SATA port, your target hard drive would likely be detected by the installer and you could actually proceed with the installation.
But of course, that wouldn't teach you how to add support for custom hardware on the fly, such as RAID SATA controller, would it, so where's the fun in that? Plus if you're reading this guide, you're trying to get an ICH controller in RAID mode supported.

In our case, if we try to iBoot our USB key, the installer won't see our target disk at all, so we need to address that on the USB key. The nice thing about our USB key procedure is that we can add AHCI support on the fly, and the producedure highlighted below should actually work for any SATA RAID controller (ICH9R, nVidia, whatever) that supports AHCI passthrough, not just ICH8R.

What's the big deal with AHCI vs RAID anyway?

As it turns out not much. The thing is, even in RAID mode, an ICH controller such as the one we use does AHCI passthrough. So why does the OSX installer see the disk in AHCI mode, but not in RAID mode?
Glad you asked. This is best illustrated by looking at the output from Linux's lspci both in BIOS AHCI mode and in BIOS RAID mode.

For the P5B Deluxe, here is what lspci -nn reports for our SATA ICH8R controller in AHCI mode:

00:1f.2 SATA controller [0106]: Intel Corporation 82801HR/HO/HH (ICH8R/DO/DH) 6 port SATA AHCI Controller [8086:2821] (rev 02)

And here is the same, but in RAID mode:

00:1f.2 RAID bus controller [0104]: Intel Corporation 82801 SATA RAID Controller [8086:2822] (rev 02)

The important things to notice is that both the class ID and the controller VID:PID have changed, from (0106, 8086:2821) to (0104, 8086:2822) respectively.
As it turns out, the 0x0106 class ID is what the Apple AHCI driver uses to find out if a SATA controller is one it should support through AHCI (well, it actually uses 0x01060100 with a 0xffffff00 mask as we will see shortly), so that's the reason why, as soon as our class ID changes to 0x0104, OSX says "Hey, I don't know how to access this type of controller!" and gives up.

That's all very well, but that doesn't tell me how to get my RAID SATA controller recognized

Patience my friend... The key to achieving anything is understanding how it actually works, which we've done, so now is the time to get to the good part.
As you may already be aware, getting hardware recognized on OSX is done through the addition or modification of kexts. A kext, which stands for Kernel Exttension is pretty much the OSX equivalent of a driver or a Linux module. Therefore, to make our RAID controller recognized by the AHCI driver Apple uses, we will need to modify the kexts from our USB installation media to trick the installer into recognizing our non-officially unsupported hardware.

The kext of interest to us is the AppleAHCIPort.kext (makes sense), that resides in /System/Library/Extensions/ (which you will often see abbreviated as S/L/E on hackintosh forums). Most specifically, what we will want to edit is the Info.plist file found in /System/Library/Extensions/AppleAHCIPort.kext/Contents.

If you look at this file you see that it contains a section:

<key>GenericAHCI</key>                                 
<dict>                                                         
        <key>CFBundeIdentifier</key>                     
        <string>com.apple.driver.AppleAHCIPort</string>  
        <key>Chipset Name</key>                          
        <string>AHCI Standard Controller</string>        
        <key>IOClass</key>                               
        <string>AppleAHCI</string>                       
        <key>IOPCIClassMatch</key>                       
        <string>0x01060100&0xffffff00</string>       
        <key>IOProbeScore</key>                          
        <integer>800</integer>                           
        <key>IOProviderClass</key>                       
        <string>IOPCIDevice</string>                     
        <key>Vendor Name</key>                           
        <string>Unknown</string>                         
</dict>

Well, well, well, is that our 0x0106 generic AHCI class right there?

Great, then we just need to change the <string>0x01060100&0xffffff00</string> part to <string>0x01060100&0xffffff00 0x01040000&0xffff0000</string> to add AHCI passthrough support to any RAID controller (if you do have an actual dedicated non SATA RAID controller on your platform, you probably shouldn't use such a blanket approach, but instead duplicate one of the later sections), save the file, reboot, and be done with it, right?

Rebuidling the kext cache

Not so fast! If you just edit and save the Info.plist on the USB key, your RAID controller still will not be detected.
The reason for that is that kexts are executed at the kernel level, so Apple doesn't simply let you load a modified kext like this. Instead, an additional step is needed, which is the rebuilding of what is known as the kext cache. This cache is the Extensions.mkext file located in your /System/Library/ directory (where the Extensions/ directory also resides).

Note that the procedure I am highlighting below can be done from an existing Snow Leopard OS, rather than on the USB key itself, but if you do so, be midnful that there's a whole business going on with regards to permissions and ownership, which, if you mount your USB key as anybody but root (not a problem when using the OSX from the key), may result in an error message such as:

AppleAHCIPort.kext is not authentic; omitting from mkext.
Authentication Failures:
   File owner/permissions are incorrect (must be root:wheel, nonwritable by group/other):
       /Volumes/Mac OS X Install DVD/System/Library/Extensions/AppleAHCIPort.kext/Contents/Info.plist

In our case, we'll run everything from the key, and the default execution level is that of the root user, so it should be fairly simple:

Open a terminal window by selecting Utilities→Terminal in the top menu (notice that we haven't had to accept any EULA to do so, so you are free to edit your OSX installation media from the embedded Terminal in any way you like)

Remount the USB key as read/write (by default it is mounted read only) by issuing the command:
```
mount -u -o rw /
```
Note: Make sure you don't close your terminal session after you do that, as Terminal will not launch if the system is already r/w (which is also the reason why we do it manually). You should be able to check that the USB file system is now mounted r/w by issuing mount

If you haven't done so already, copy the nano text editor, which you should have copied into the root directory to /usr/bin with the command:
cp /nano /usr/bin
Or you can just leave it in root and issue /nano whenever you need to edit a file.

Navigate to the Info.plist file we want to edit with cd /System/Library/Extensions/AppleAHCIPort.kext/Contents/ and run nano nano Info.plist

Change the <string>0x01060100&0xffffff00</string> line in the <key>GenericAHCI</key> section to <string>0x01060100&0xffffff00 0x01040000&0xffff0000</string>

Save the file (Ctrl-X then 'y' to save it)

navigate to the /System/Library/ directory with cd /System/Library/. This is the directory that contains the Extensions.mkext file

Rebuild the kext cache with:
```
kextcache -v 1 -t -l -a i386 -a x86_64 -m Extensions.mkext Extensions
```
It should complete without errors (or a benign warning about the JMicronATA.kext) but if there are any issues reported about the AppleAHCIPort.kext, they should be explicitly listed.

Close terminal and leave the installer (Mac OS X Installer→Quit Mac OS X Installer) then select Restart. You must restart the installation process completely (i.e. reboot) for our changes to be applied.

With all the above completed, on the second iBoot/USB installer run, you should now see all hard drives recognized, including the ones connected in a SATA connector in RAID mode, with the ability to use them as installation destination. Neat!

Before shouting "victory" though, you should be aware that:

When OSX installs, it will extract and install its own unmodified AppleAHCIPort kext (from the Essentials.pkg found in System/Installation/Packages/ directory on the installation media, so what we have done will only work during installation, but the installed OSX will not actually be able to see its disk and we'll have to work around that - bummer!

iBoot offers the option to ignore the cache during bootup (press the down key on the OS you want to boot, and then select "Boot Ignore Caches"), so we probably could have gotten away with simply modifying the Info.plist file and used that iBoot option instead of going through this whole cache rebuilding exercise...

Anyway, in the next installement, we'll go through the actual OSX installation process. Stay tuned...

Installing OSX (Snow Leopard) + Linux (Slackware 13.37 x86_64) on a GUID/GPT disk, with Software RAID enabled (ICH8R) - Part 1

2011-05-23T22:31:00.014+01:00

Now that's quite a mouthful of a title. And considering that there is much to discuss, I will break this whole thing into a multipart post, so here is the first part of our series.

Introduction & Requirements

First of all, I gotta start with the disappointing news that, despite what you might anticipate from the title, this is not a guide to setup motherboard RAID (a.k.a. fakeRAID) support in either OSX or Linux (though the later is no biggie - see previous posts here for various options). The mention of RAID here is to indicate that we want to do an OSX and Linux that coexists nicely with a previous fakeRAID installation, by NOT requiring you to disable RAID settings in BIOS if they are already enabled.

In practice here, the goal is to add OSX and Linux on a separate non-RAID disk to an existing Windows 7 installation that was set to use fakeRAID (RAID 1 from ICH8R), without touching the BIOS settings.

Basically, what happened is that I recently upgraded my good old Asus P5B Deluxe based rig to a dual Samsung F3 1TB, installed Windows 7 ontop of the ICH8R RAID 1 array which I created in BIOS as a result, and since that left me with a bunch of unused SATA drives, I thought I would put one of these to good use by installing both OSX and Slackware on it (that Slackware 13.37 sure IS nice!), and use it to boot the whole lot. Sounds easy, but it actually presents a nice set of challenges, some of which include:

Adding extra drivers to a vanilla OSX installation media, so that it recognizes our AHCI controller even in RAID mode
Allow LILO (or GRUB) to boot OSes other than the main Linux system on a GPT/GUID partitioned disk (with a BIOS that is NOT EFI)
Installing OSX Snow Leopard from scratch, including updates and necessary drivers for your rig, but WITHOUT having to spend one's life endlessly browsing the OSX86 forums. Yes, the effort is much appreciated guys, but I have better things to do then become an expert on kexts, Chameleon, DSDT, iBoot and whatnot.

My specs:

Asus P5B Deluxe with 2x 1TB SATA HDDs in RAID mode (ICH8R) and 1x 320GB HDD (non RAID)
nVidia GPU (7950GT)
PS/2 keyboard, USB mouse + IDE DVD (the DVD is only going to be used for iBoot, so SATA or IDE won't matter one bit)

Requirements:

OSX Snow Leopard official installation DVD or .dmg image (build 10A421A)
One 8 GB (or bigger) USB key
An existing Snow Leopard installation. Yeah, I know it sucks, but hey.

That last item means either you need to create a VMWare image from the DVD media, or borrow a friend's Mac, but we want to create a USB OSX installation medium, which we can customize so there's little way to bypass that requirement. If you really don't have access to a preexisting OSX system, it's probably possible to create an USB key from Linux from the OSX installation DVD, but I haven't tried it so I can't comment on that. Besides, a VMWare image with Snow Leopard shouldn't be that hard to find.

My kingdom for AHCI? Surely you are jesting!

Question: What do all the hackintosh installation guides you find on the internet have in common?
Answer: They all begin with "You MUST change your BIOS settings and set your SATA controller to AHCI before you begin the installation"

Of course, this is utter nonsense. If a guide begins with "You MUST change your BIOS settings to something potentially incompatible with what you want, and risk destroying existing data as a result" (by disabling RAID mode from ICHR), then it's a lousy guide and you should stop right there. For having been a fakeRAID/motherboard RAID user for some years now, with both Intel and nVidia solutions, I'll tell you frankly: all those who say "Don't use fakeRAID: buy a dedicated RAID controller instead" are idiots. First of all, money to buy a separate controller doesn't grow on tree and secondly, with all these cores lying around, the idea that one should spend extra, when all they want is add fault tolerance to consumer grade hardware is ludicrous. We're not building an enterprise grade server here dammit! Besides, most of these solutions are rather well supported in Linux now, which means that if you screw up your fakeRAID array, you may just be able to do something about it. With a hardware proprietary non reverse engineered RAID solution on the other hand, you might not be so lucky...

Therefore, the hell with changing your BIOS settings to AHCI! If Linux doesn't bat an eyelid about accessing SATA HDDs with an ICH controller in RAID mode, why should OSX be the exception? Therefore, we'll keep our ICH BIOS settings to RAID, thank you very much, and ignore all this AHCI brainwashing.

Now, since the OSX installation media has not been designed to support ICH8 in RAID mode, if you try something like iBoot and an unmodified OSX installation media, then unless your HD is plugged on a JMicron SATA controller (which is an actual possibility on the P5B Deluxe), the OSX installer will not see it. This basically means we're gonna have to modify the OSX installation system so that it sees our ICH8R controller as AHCI ICH8 and that's also where doing the install from an 8GB (or more) USB key, that we can modify on the fly, will come real handy.

But first, we gotta create the base vanilla installer on our key.

Creating a vanilla OSX installation USB key

To do that, you need to be in your existing OSX system (VMWare, soon-to-be-ex-friend's machine, etc.). I'm going to assume that you have the vanilla OSX disk as a .dmg. The steps to create the key then are as follow:

Plug in your USB key. Don't worry if it automounts.
Open the OSX disk utility (Applications -> Utilities ->Disk Utilities). You should see something like "8.02 GB Generic STORAGE DEVICE Media" or something, as your USB volume
Click on the "Partition" button on the left, to access the partition tab, then click on the "Current" Volume Scheme button so that you get a dropdown, and select 1 partition
In options, under the partition diagram, make sure you select "GUID Partition Table". For the Format, you select "Mac OS Extended" (though it shouldn't matter)
Click Apply to partition & format the USB key
Once formatting is complete, select the Restore button
Now double click on your OSX .dmg image to mount it (or insert your OSX DVD). A "Mac OS X Install DVD" icon should appear on your desktop
Drag that icon onto the Source field of the Restore parameters in Disk Utility until a green '+' sign appears, and then drop it there
For the Destination field, just drag and drop the "Untitled 1" partition, or whatever you called it, from your newly formatted USB stick
Click the "Restore" button. You will be prompted for the system's administrative password.

For those who want to see it for themselves, the whole process of creating an OSX installation USB key is demonstrated by the first part of the video below (credits go to stellarola - you can ignore the Stella Magic part that comes afterwards):

The restore process will take a few minutes (15-30 mins), but, once completed, will give you the ability to install OSX from USB using iBoot.

Installing a text editor

Because we're going to do it the hard way, a must have for our USB key is a text editor, so that we can tune the USB key to our needs during the installation process (more about this in the next post). As a UNIX veteran, I swear by vi, but I do understand that not everybody is confortable with vi sequences, so to alleviate the editing process, we'll go with the more intuitive GNU nano.

One might wonder why there is no default editor on the vanilla OSX installation media, but the answer is that Apple obviously considered an installer to be read-only affair, so it made little sense adding vi or nano to the installation utilities. On the other hand, we will very much require the ability to edit files on USB, so we need a text editor for OSX. Of course, since GNU nano is GPL and therefore, anybody can legally publish a nano OSX binary to copy onto an USB key, I'm not going think twice about putting the one I conveniently recompiled from source (using Xcode 3.2) and that I am putting online here.

Once downloaded, just drag and drop the "nano" file onto your USB stick folder (where you'll see the OSX installation paraphernalia). As long as it's on the key, we're good - no need to worry about placing it into bin/ or usr/bin/ just yet. Once it's copied, you can eject the USB, as at this stage, we won't need to access another OSX system: we'll just do everything we need from USB.

iBoot

Finally, the last piece we need for the installation is the latest iBoot CD (bugmenot). And while on the iBoot download page, you might as well download MultiBeast, as it will come handy to finalize our installation.

The iBoot zip basically contains an ISO which you should burn to a CD. Note that if you can't use a CD, you should be able to use that ISO image to produce an alternate iBoot loader (eg use a second USB key or TFTP), but I'm not going to detail that.

Why encrypted firmware is bad

2011-01-21T12:14:00.003Z

A simple example will suffice.

Let's say I own a digital camera, or a device that contains a digital camera, that uses encrypted firmware.

Now let's say that this firmware has been written in such way that, at regular intervals, and without any form of notification to the user (like a LED blinking or a tell-tale shutter noise), it takes low resolution pictures of what it sees, and stores these unrequited pictures into its embedded storage (which too would be encrypted).

Then, when the user takes a normal high res picture using the digital camera, the firmware adds "noise" to it that contains encrypted image(s) from the hidden low-res pictures it has in memory.

With hi-res image files in the MB range or even better: digital video, it has become exceedingly easy to add "meaningful noise" and play a little steganography on digital photography. There are also ways to ensure sure that an useful data payload can still be recovered, even if the user downsizes the images.

Blissfully unaware of this, the user then uploads some of these high res picture to a public website, as one does. Then malicious entities/government agencies can just run a search on EXIF (if the user didn't remove this data, but even then, with the proper resources, parsing images all day to check for known steganographic payload is not that big a feat) and spy on you at length, provided you keep sharing pictures with friends, etc, and without your knowledge...

Of course, this kind of far fetched scenario would never happen... just like printer manufacturers would never add hidden marks to every page printed, that would uniquely identify what printer (and by extension who) printed some data.

Formatting a PSP Memory Stick for use with a Pandora battery in Linux

2010-08-13T17:44:00.005+01:00

Always a pain to do, and nobody seems to provide the files I want, so I'll just provide my own files (using ipl_ms.bin => no frills, just normal boot) and a short script.

Once extracted, just run something like:

root@sheeva:~/pandora# ./format_ms.sh /dev/sda
+ dd if=/dev/zero of=/dev/sda bs=512 count=32
32+0 records in
32+0 records out
16384 bytes (16 kB) copied, 0.00699201 s, 2.3 MB/s
+ parted /dev/sda
GNU Parted 1.8.8
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos
(parted) mkpartfs primary fat16 32s -1s
(parted) set 1 boot on
(parted) set 1 lba off
(parted) u s
(parted) p
Model: Generic STORAGE DEVICE (scsi)
Disk /dev/sda: 3995648s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start  End       Size      Type     File system  Flags
 1      32s    3995647s  3995616s  primary  fat16        boot

(parted) q
Information: You may need to update /etc/fstab.

+ dd if=ipl_ms.bin of=/dev/sda bs=512 seek=16
4+1 records in
4+1 records out
2288 bytes (2.3 kB) copied, 0.00179176 s, 1.3 MB/s
+ mount /dev/sda1 tmp_mnt
+ tar -C tmp_mnt -xvf ms.tar
ISO/
ISO/VIDEO/
MEMSTICK.IND
MP_ROOT/
MP_ROOT/100MNV01/
MP_ROOT/101ANV01/
MSTK_PRO.IND
MUSIC/
PICTURE/
PSP/
PSP/GAME150/
PSP/GAME5XX/
PSP/GAME/
PSP/SAVEDATA/
PSP/COMMON/
PSP/SYSTEM/
PSP/THEME/
PSP/RSSCH/
PSP/RSSCH/IMPORT/
VIDEO/
seplugins/
+ umount tmp_mnt
+ sync

That darn iBFT iSCSI Windows installation error

2010-08-09T12:01:00.019+01:00

If you ended up here, it's probably because you too tried to install Windows 7 or Vista on an iSCSI bootable disk using gPXE and, even though Windows setup could see your disk alright, you got one of the following errors:

"Windows cannot be installed to this disk. Setup does not support configuration of or installation to disks connected through a USB or IEEE 1394 port" (Vista)

"Windows cannot be installed to Disk <#> Partition <#>. (Show details)" -> "Windows cannot be installed to this disk. iSCSI deployment is disabled since no NICs referenced in the iBFT can be resolved to actual NT-visible devices. Windows cannot be installed to this disk. This computer's hardware may not support booting to this disk. Even if you're probably smart enough to know what you're doing, and we could definitely let you install to this disk to sort booting later, we're going to be asses about it and prevent you from overriding the idiotic setup decisions we made (by the way, did we mention the Windows recovery partition yet?). Why? Because we're Microsoft and screw you, that's why." (Windows 7)

OK. Let's forget about Microsoft's stupid decisions for a while, and attempt to work within them, to figure out how we can address the issue.

Firs of all, if you see the message above, then I can guarantee that, no matter how properly you think you did setup your iSCSI PXE boot, you screwed up something and your iSCSI boot sequence is wrong.
And yes, getting an iSCSI boot error on a blank disk is to be expected, but NO, not all the errors you see during gPXE iSCSI boot can be safely ignored (if you manage to see them at all, but we'll come to that). There's good error and there's bad error.

The first thing I'll point out, if you're like me and thought you could get your dchp/tftp server to:

Supply the iSCSI disk boot parameter (that dhcp-option=net:gpxe,17,"iscsi:192.150.23.3::3260:2:sheeva:disk1" line or similar, along with the keep-san option)
Attempt to boot from it and
If that fails, fallback to executing WinPE/pxleinux to launch a WinPE installation image

is that such a scheme just won't work. Unless you're fiddling with the gPXE scripting options (and even then), you can only have either

Boot from iSCSI then fail and hands things over back to BIOS, or,
If a boot image is specified, ignore the iSCSI options provided by the dhcp server altogether and just boot from that image.

You can't just have the dhcp/tftp server alone tell gPXE: "try to boot to iSCSI and if that fails, boot from something else while keeping the iSCSI boot options", to boot WinPE for installation onto a blank iSCSI disk for instance. No siree. If you tried that, well, that was your first mistake. Not to say that this can't be achieved at all (we'll see how to do just that from the commandline below, and, in a next post, I'll try to show you how to do it automatically as well), but that can only be achieved outside of the dhcp/tftp options.

In short, if you're using dnsmasq and with something like:

dhcp-match=gpxe,175   # tags the request with net:gpxe if gPXE was supplied
dhcp-option=175,8:1:1 # turn on the keep-san option (allows installation)
dhcp-option=net:gpxe,17,"iscsi:192.150.23.3::3260:2:sheeva:disk1"
dhcp-boot=net:#gpxe,pxelinux.0       # if NOT (#) gPXE, use pxelinux.0
dhcp-boot=net:gpxe,Boot/startrom.n12 # if gPXE, use WinPE

Then, when WinPE boots, it will not have any of the options that you think gPXE should have fed it with regards to the iSCSI boot disk. Especially, the "dhcp-option=net:gpxe,17," option will be completely ignored. Yeah, that makes as much sense to me to as anybody else, but that's how gPXE works for now.

And that's also the reason why, in most of the guides you see, they'll tell you to first try to boot from an unbootable iSCSI disk with gPXE, let it fail and then use BIOS fallback to boot from an installation CD or DVD. Again, simply chaining WinPE in there from PXE does not work without additional effort that none of these guides provide.

Also, and this is the most important part if you want Windows install to accept your iSCSI disk as bootable, as long as you do not see the following lines during boot:

Booting from root path "<your iSCSI path>"

Registered as BIOS drive 0x80
Booting from BIOS drive 0x80
Boot failed
Preserving connection to SAN disk

Then it's game over, plain and simple.

Granted, those line may be hard to spot at during boot, when gPXE will hand things back over to the BIOS on failure (which it should do, if you followed what I said above), as those darn BIOS makers forgot that the Pause key we have on our keyboards could be put for some good use, but if you try a few times, and you don't see any mention of a BIOS drive 0x80, Windows will simply not see your iSCSI driver as bootable, simple as this.

For your reference, here's a screenshot from a VMWare diskless machine that illustrates what you should see when gPXE executes:

As long as you see the lines I highlighted above, after the iSCSI boot attempt, whatever error is thrown out will come from the iSCSI disk itself, rather than your boot process, so you can ignore it. But if you don't see the "Registered as BIOS drive" line from gPXE however, you should pay very close attention to the iSCSI error you get.

So, of course, now your question is: "I'm not seeing these lines (or they're too fast for me to see). How then can I validate that my iSCSI target is good, and that it can be used for installation with gPXE?"

Well, duh, through the gPXE commandline of course, which you can enter with Ctrl-B at boottime. Gotta wish proprietary PXE was as easy to troubleshoot for power users as gPXE is. But you're in a hurry and don't want to learn about the whole gPXE/DHCP/TFTP internals, so I'll cut down to the chase. The sequence of command you are after:

dhcp net0
set keep-san 1
sanboot iscsi:<iscsi server ip>::<iscsi port>:<iscsi lun>:<iscsi target id>
# and if the above line works and you want to boot to WinPE for instance, you could
chain tftp://<server ip>/Boot/startrom.n12

dhcp net0 initializes DHCP and allows you to communicate with the server (for tftp, etc)
then the keep-san option is to ensure that Windows can see the iSCSI disk as bootable, which of course is the feature you're after
finally the sanboot line is the one that will tell you if something is wrong with your iSCSI access.

But first, let's see an example of what happens when everything works as expected (for an uninitialized disk):

Here, we have the Registered as BIOS drive and the Preserving connection lines so we're good. You might also want to note that I am specifically specifying that I'm using port 3260 (default for iSCSI) and that my device is on LUN2 (very non default).

Now, let's see some common errors:

0x2c0d603b is usually an indication that your iSCSI path is wrong. In the case above, I used the non-existing disk0 instead of disk1 for the target part.

Ah, 0x1d704039 (and now, aren't you glad you found this page)...
Yes, this is an error you should not get, even with a non bootable iSCSI disk. And yes, I agree, that an I/O error is precisely what you'd expect from a non-bootable disk, but actually, that I/O error is unrelated to the disk being bootable or not. On the other hand, it has very much to do with trying to use an iSCSI device that cannot be used as a disk, which, if you are using Linux tgtd/tgtadm is exactly what you're going to get if you're leaving the sequence of four columns as is (::::) because that means that LUN 0 will be used, and LUN 0 is reserved by tgtd for the virtual controller.
In short, if you're using tgtd, your actual device LUNs start at 1, and if you're keeping the options part as '::::', then the default of LUN 0 will be used, so you're not actually accessing your disk!
This is why I'm using iscsi:192.150.23.3::3260:2:sheeva:disk1 in the line that works, because I'm trying to access the second disk I created on that specific target. Even if that was the very first disk I created with tgtadm, I would have to add 1 for the LUN in the line above, because the default of 0 is not a disk.

Then, other errors you might get are 0x0b8080a0 (Operation cancelled) or 0x2e852001 (Exec Format Error), but these should occur after you get the "Registered as BIOS drive" line, so you should be able to safely ignore them. For other errors, google is your friend.

So, to summarize, if Windows doesn't like your iSCSI boot device, it's probably because, despite what you think, gPXE didn't find anything it could use as a bootable disk, and to find out why, you should try to boot from it using the gPXE low level commands.

In a next instalment, we'll see how we can create a nice iSCSI aware WinPE image, that we can launch from PXE, for all of our installation needs, and how we can solve the problem of automating WinPE fallback from a non bootable iSCSI disk, as well as how we can use pxelinux to boot from multiple iSCSI disk.

chrooted ssh & sftp on Slackware

2010-06-25T15:31:00.006+01:00

Since the SheevaPlug makes a nice server for one to share files online securedly, while allowing you to see exactly who is accessing them, today's exercise is to setup sshd so that selected people can SFTP into it, while only seeing what you want them to see.

Now, you'll find plenty of articles on how to do that, some of them very well made, but what they won't tell is how to sort things out when stuff's not exactly working as it should be.

First of all, this is what you want to have in your sshd_config:

# Use a non obvious port for outside connections.
# You could also do port translation on your gateway or something
# but, just so you see how it's done:
Port 22
Port 1234

Default options should be super limiting: no X11/TCP forwarding, no pasword auth, no root logon, etc) and then you use the very convenient Match tag to setup the allowed remote users, as well as the less restrictive options for your own network. Thus:

# only "remote_user" can logon from the outside network, in a chrooted env
Match User remote_user
     PasswordAuthentication yes
     ChrootDirectory /home/chroot
# connections from inside are OK
Match Address 192.168.1.0/24
     PasswordAuthentication yes
     PermitRootLogin yes
     AllowTCPForwarding yes
     X11Forwarding yes

Then I'll assume that you pretty much follow the link I gave above to setup your /home/chroot directory and create the "remote_user" guy.

First bad surprise:

"cannot run command `/bin/bash': No such file or directory"

WTF? But I did copy bash and the script took care of my libraries.
Unfortunately, nope, the script did not take care of all the libs, and instead of reporting "library not found", the error message does not help.

Now, if you go:

# ldd /bin/bash
     libtermcap.so.2 => /lib/libtermcap.so.2 (0x4004b000)
     libdl.so.2 => /lib/libdl.so.2 (0x40056000)
     libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40061000)
     libc.so.6 => /lib/libc.so.6 (0x40075000)
     /lib/ld-linux.so.3 (0x40000000)

and check your libraries again, you'll probably find that ld-linux.so.3 was not copied. If you do just that, you should find that you can logon to the chrooted environment at last. Yay!

But then you want to add sftp. To do that, you *must* have the sftp-server and the lib dependencies in your chrooted environment as well.

For Slackware, that means you need to create a /home/chroot/usr/libexec/sftp-server (if you don't know which sftp-server to pick, check the line "Subsystem sftp" in your sshd_config) and once again, copy all the library files (or use the script).

But then, disaster strikes: whenever you try to logon with sftp, you get your connection closed. If you look at the sftp debug log, you'll see something like:

debug1: Sending subsystem: sftp
debug2: channel 0: request subsystem confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 131072
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: rcvd close
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cfd -1)

debug3: channel 0: close_fds r -1 w -1 e 6 c -1
debug1: fd 0 clearing O_NONBLOCK
debug3: fd 1 is not O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.3 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 1
Connection closed

Well, to cut a long story short, if you're seeing this, the problem is likely that you don't have rw permission on /dev/null (and possibly /dev/zero). Just make sure you issue a chmod 777 /home/chroot/dev/null and that should be the end of it.

Resync'ing a RAID 1 array

2010-02-24T13:37:00.008Z

Can come handy when attempting to correct unreadable sectors on a RAID disk. Eg. if your RAID1 array is built around /dev/sda3 and /dev/sdb3 and you had unreadable sectors reported by SMART on /dev/sda (Current_Pending_Sector), but got no error during the SMART extended self test, you might want to resync the /dev/sda partitions on /dev/sdb as follows:

mdadm --fail /dev/md2 /dev/sda3
mdadm --remove /dev/md2
/dev/sda3
mdadm --add /dev/md2 /dev/sda3

Changing the NTP update interval in Windows

2009-12-31T12:54:00.002Z

Default is 604800 seconds (7 days), which is way way too long. On my machine, I'm getting clock skews of close to a minute as a result, and that's not acceptable when you're swapping files around for compilation for instance.

To set this value to a more reasonable interval, you need to update the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\NtpClient\SpecialPollInterval

Windows 7, I hardly knew ye...

2009-12-15T11:03:00.008Z

Blah, blah, blah, Windows 7 is great, blah, blah...

Oh, you were actually expecting something in that range? Not on this blog I'm afraid.
When Windows 7 consistently freezes within 4 hours of a fresh install (happened on Vista too, so I don't think it's a Win 7 issue only. Past that initial freeze the system is stable enough though), needs a full reinstall less than one month after it, and putting the computer to sleep or rebooting is akin to Russian roulette (will it boot again, or will just remain frozen forever and require, not only a hard reset, but a complete PSU unplug? - heck, I have a hackintosh OS-X working better than that on hardware that was never meant to be supported by Apple - go figure!)

And now, once again, it's time for a clean Windows 7 reinstall, which means reinstalling all the apps, and having to do that every few months to keep the hardware in running order (because I actually DO something with my machine you know, like installing drivers by the truckload for development purposes - it's not just for internet and multimedia) is getting a bit old.

Today's tip then is then is how to avoid reinstalling the WinDDK, when you still have the files (in E:\WinDDK\7600.16385.0\ for instance) and all you are interested are the build environment shortcuts.
Windows 7 x64 free build environment shortcut:

C:\Windows\System32\cmd.exe /k E:\WinDDK\7600.16385.0\bin\setenv.bat E:\WinDDK\7600.16385.0\ fre x64 WIN7 no_oacr

Same thing for x86:

C:\Windows\System32\cmd.exe /k E:\WinDDK\7600.16385.0\bin\setenv.bat E:\WinDDK\7600.16385.0\ fre x86 WIN7 no_oacr

And for reference, setenv usage:

Usage: "setenv  [fre|chk] [64|x64] [WIN7|WLH|WXP|WNET] [bscmake] [no_oacr] [separate_object_root]"

HDMI-CEC is SLOOOOOOOOOOOOOW!!!

2009-11-05T16:28:00.007Z

Well, like many, I was convinced that HDMI-CEC was the wave of the future with regards to AV automation, until I actually read the specs of the thing (in "Supplement 1 Consumer Electronics Control (CEC)" of the HDMI Specifications version 1.3a).
The interesting bits are to be found in "CEC 5 Signaling and Bit Timings" and "CEC 6 Frame Description", where, if you do the maths, you actually find out that the shortest time it takes to transfer a single 8 bit packet (start bit + 10 bit header + 10 bit packet) is 52.5 ms. And if you were to transfer 32 bits of data, you get close to 125 ms, or, in other words, no more than eight 32 bit data packets per second.
Heck, I can mash a remote button faster than that!

What this means really is that. no matter how cool HDMI-CEC might look, with a max data rate of only 41 bytes per second, future proof it ain't.
Thinking of transferring a GUI menu content from one device to another using CEC? Won't happen.
Thinking of transferring a reasonable amount of text in a short time? Huh-huh.

For a format designed in 2002 to be more than 20 times slower than what has been the de-facto lowest speed for any form of serial data communication (9600 bauds) for more than 30 years now, you really got to be kidding us.
And it's not like either the cable or devices connected can't support high transmission speed (it's HDMI - both the cable, and the devices it connects were designed for high speed!).

So really there you have it. On one hand, a handful of transmission lines that can transfer at least half a Gigabyte of data per second, and in the same cable, a puny line that will barely transfer 1/1,000,000th of it, because of our regular corporate overlords' lack of vision (guys: if you're going to borrow what SCART has been doing for more than 2 decades, at least bring it up to speed).

Oh the irony!

Creating a Visual Studio 2008 application that uses Cairo

2009-10-29T13:08:00.014Z

One day or another, you'll want to produce quick 2D data with your code for visualization, and after looking through hordes of libraries that do not satisfy your needs at all (listen, it's not that hard: I don't want graphing, I don't want 3D, and I certainly don't want proprietary closed source paying software! I just want something that gives me a simple canvas to draw elementary stuff like text fields or simple graphics using 2D coordinates), you'll settle down on Cairo as the best trade-off for quick and easy generic 2D output. If it's used by Firefox for SVG output, it certainly should satisfy our needs.

Unfortunately, or should I say, as usual, whenever you want a nice step by step tutorial on how to get you started with using Cairo on Windows, you'll get nothing but a handful of incomplete snippets here and there. This short step by step tutorial attempts to remedy that.

For this exercise, we'll just create a C console application that outputs "Hello, World" to a PNG file using Cairo using Visual Studio 2008.

Create a new Win32 Console Application in Visual Studio. Let's call it cairo_test. And since we don't wanna get bogged down by Microsoft's crap on a simple hello world app, on Application Settings -> Additional Options, make sure you check "Empty Project".

Create a new source file - let's call it main.c - and fill it with the following content (which I picked up from here):

#define LIBCAIRO_EXPORTS

#include <cairo/cairo.h>

int main(int argc, char** argv)
{
cairo_surface_t *surface;
cairo_t *cr;

surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32, 240, 80);
cr = cairo_create (surface);

cairo_select_font_face (cr, "serif", CAIRO_FONT_SLANT_NORMAL, CAIRO_FONT_WEIGHT_BOLD);
cairo_set_font_size (cr, 32.0);
cairo_set_source_rgb (cr, 0.0, 0.0, 1.0);
cairo_move_to (cr, 10.0, 50.0);
cairo_show_text (cr, "Hello, World");
cairo_destroy (cr);
cairo_surface_write_to_png (surface, "hello.png");
cairo_surface_destroy (surface);

return 0;
}

Don't worry about the "#include <cairo.h>" for now, we'll sort it out in a second

Download the latest Cairo Dev package files for Windows from http://www.gtk.org/download.html by picking up either Windows 32 or 64. At the time of this article, the latest Dev is cairo 1.8.8.
Extract the package directories "lib" and "include" at the root of your Visual Studio project. You can safely ignore the other directories from the archive.
Change the Active configuration if needed and right click on your project in the Solution Explorer panel to access the properties page.
- In Configuration Properties -> C/C++ -> Additional Include Directories, create a new entry and point it to "<your project root>\include". Be mindful that there is a cairo subdirectory there, which is why we used cairo/cairo.h in our source. Just make sure the source and your include paths match.
- In Configuration Properties -> Linker -> Input -> Additional Dependencies, type "cairo.lib"
- In Configuration Properties -> Linker -> Gener al -> Additional Library Directories, create a new entry and point it to the "<your project root>\lib" directory you extracted above. (Oh, and why oh why are the Additional Dependencies and Additional Library Directories on 2 different pages Microsoft?!? Where's the twisted UI logic behind that?)
Try to compile your project. It should complete without errors. Note that if you picked up the Windows 64 libraries, you MUST create a new x64 configuration, or the process will fail with "fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86'". Just follow these guidelines if you don't know how to do that.
Bet you didn't wait and tried to run your executable already. And of course, you got the DLL not found errors. Why of course, now you need to install the bunch of DLLs Cairo needs to be happy. Basically, what you should do is pick up the DLLs from ALL the binary packages below (still provided by the GTK+ Windows 32 or Windows 64 Project binaries) and extract them into the Release or Debug directory that contains your executable. For all of the archives, you j ust need to extract the DLL - all the other files are irrelevant:
- cairo Binaries (yes you need the binaries package too, as the Dev one doesn't contain the DLL) -> libcairo-2.dll
- zlib Binaries -> zlib1.dll
- libpng Binaries -> libpng12-0.dll
- Freetype Binaries -> freetype6.dll
- FontConfig Binaries -> libfontconfig-1.dll
- expat Binaries -> libexpat-1.dll
If you're gonna produce JPEG or TIFF images with your application, you probably want to install those DLLs too, but you already guessed that.
Now you can actually run your test program. It should produce an "hello.png" file that looks like the one below:

Alrighty then. Now you can get going and visualize the hell out of whatever ground-breaking application you've been thinking of!

Formatting node titles in drupal

2009-10-21T17:17:00.010+01:00

Bit disappointed in Drupal. Of course it's a great tool (and it's GPL), but they probably won't be 'there' till version 8 at best with regards to full fledged & intuitive customization.
Not that I really mind modifying the PHP code anyway, but I'd rather create actual content instead of doing that.

So, of course you have installed the cck module to create fields, and it's oh so convenient (although customizing the field formatting can be a bit of a pain there too), but unfortunately, the title field for a node does not fall under the cck realm, and thus you don't have means of applying any of the default or custom formatters you might be using with cck. Bummer!

Let's say you have created a new content type, with type 'my_ct' then (would be listed along with 'book', 'page', 'story'), and you are using a numeric title field (again, with no possibility of setting a content type for node titles and check input, Drupal is quite limited). Now you want to display your title in a custom format. Eg. if your node title value is '12', you want your displayed title to be "My_Content_000012", with leading zeroes.
For that, assuming you are using the default garland theme of Drupal v.6, edit the file drupal/themes/garland/template.php and in "function phptemplate_preprocess_page(&$vars)" add the following:

  // Custom format for my_ct node titles
 if ($vars['node']->type == 'my_ct') {
   $value = $vars['node']->title;
   // Don't apply formatting unless numeric
   if (!is_numeric($value))
     $vars['title'] = $value;
   else
     $vars['title'] = sprintf("My_Content_%06d", $value);
 }

Now, if you want better customization or input validation for titles, you can probably get started on that with the following article.

Don't use Nero, period

2009-10-19T14:34:00.010+01:00

Growing tired of Nero's inability to handle cue sheets (wanna make Nero crash? Just pick up a cue sheet here and there). This program has so many drawbacks, it's a wonder they still manage to sell it (yes, Ahead, it is normal in 2009 for a computer to become totally unusable for about 30 seconds whenever you insert a CD and Nero is running - background poll-wah?).

The answer: use cdrecord and get back the time and coaster you'd have gotten with Nero.
To burn a CDDA with text from a CUE sheet using cdrecord:

cdrecord dev=1,0,0 -eject - copy -v -text -useinfo -dao cuefile=<some file>.cue

Now, once I get a 64 bit, non cygwin1.dll dependent version of cdrecord in MinGW, I'll be a happy man.

Task Manager Windows Shortcut

2009-10-15T15:51:00.003+01:00

Found that one totally by accident (and yes, sometimes it doesn't hurt checking the official Microsoft documentation): <Ctrl> <Shift> <Esc> = bring up the Task Manager

Now, give me a shortcut to bring up the Device Manager (well, you can go there in 2 steps, through <Windows <Pause/Break> (= System Information) -> Device Manager, but still...

Oh, and of course, you want to replace the default Task Manager with Process Explorer, as the shortcut works for it too.

Compiling OpenOCD on Windows 64 systems

2009-10-13T13:58:00.008+01:00

This time with libftdi/libusb - just follow the leader. ;)

Or, if you just want my latest (development) installable zip of OpenOCD for Windows 7 64 / Vista 64, try here. Just extract, follow the instructions, and you should have OpenOCD 0.3.0 running on Windows 64 bit in no time.

Took a little while to get this stuff working, because of a very nasty libFTDI bug, as well as the usual hurdle of driver signing for Windows 64. I think the FSF should establish a free driver signing program for well established GPL projects, like OpenOCD, so that we get "Microsoft Approved" signed drivers. That would foil Redmond's obvious attempts to oust OpenSource from Vista 64 / Windows 7 64 (although they'd have to walk into the scheme and pay Microsoft/Verisign for the signing privilege).

There's got to be a way to restore freedom back to the users of Windows OS!

Installing Slackware on the SheevaPlug

2009-10-09T16:05:00.033+01:00

And so, finally, we come to the final part of our series regarding the setting up of our SheevaPlug with Linux. Thanks to the good guys at armedslack, we will have the best Linux distro ever, running on one of the coolest little devices around.

For this setup, we will install Slackware on an 8 GB High speed SDIO MMC card, and leave the NAND flash untouched with its updated Ubuntu version. This will also suppose that you have a working kernel that supports ext4 and all the features you want already installed on the flash (if not, refer to this post to to set it up). Now, we are going to deviate a little bit from the "official" armedslack intallation guide for the SheevaPlug, as we are not going to use TFTP for our packages but instead read them directly from an USB device.

And so, without further ado:

Download the packages from armedslack.org. The only directories you really need are the "slackware", "isolinux" and "kernels" directory from ftp://ftp.armedslack.org/armedslack/armedslack-current/. If you're downloading from a Linux system, one way to do it, as advertised in the armedslack installation readme is:
```
rsync --exclude '*/source/*' --delete -Pavv ftp.armedslack.org::armedslack/armedslack-current .
```
(NB: If you use Filezilla to download from FTP, please note that it seems to default to ASCII mode instead of binary, so don't be surprised if your files are corrupted!)
This amounts to about 2 GB of files, so it will take a while, but the filesystem used for the target shouldn't matter much, as long as it is accessible from the default armedslack kernel. So if you have a FAT formatted USB stick lying around, with more than 3 GB space available, you should probably use that.
I also have to point out that we are downloading slackware-current, which contains an up-to-date version of the packages, but is different from a dot release, so you might see some versioning differences between current and what you get on the official x86 dot releases.
For the remainder of this guide, I will assume that you have the directories (slackware/, isolinux/ and kernels/) in an "armedslack/" directory at the root of a FAT formatted USB device. Note that if you're not using a FAT formatted device, you will need to copy the kernel and initrd image on a separate device that can be read from U-boot

Alright, time to reboot the system, with your USB and MMC card plugged in and enter the Marvell>> U-boot prompt through the serial console. There, you need to issue the following:

Marvell>> usb start

(Re)start USB...

USB:   scanning bus for devices... 2 USB Device(s) found

scanning bus for storage devices... 1 Storage Device(s) found

Marvell>> fatload usb 0 0x00800000 armedslack/kernels/kirkwood/uImage-kirkwood

reading armedslack/kernels/kirkwood/uImage-kirkwood

...

................................................................................



1887248 bytes read

Marvell>> fatload usb 0 0x01100000 armedslack/isolinux/uinitrd-kirkwood.img

reading armedslack/isolinux/uinitrd-kirkwood.img

...

................................................................................



10985095 bytes read

Marvell>> setenv bootargs console=ttyS0,115200 nodhcp kbd=uk root=/dev/ram rw;bootm 0x00800000 0x01100000

If all is well, you should be seeing your usual Slackware installation screen.
Please note that because of the "kbd=uk" line above, your keyboard will be automatically set to uk, so change it accordingly.
Also, if you want to use a network session rather than the serial console, you should issue:

dhcpcd eth0

/etc/rc.d/rc.dropbear start

Then you can find your IP address through ifconfig and ssh to it.

If needed, you should use fdisk to partition your MMC device:
```
fdisk /dev/mmcblk0
```
On my system, I created a 512MB /dev/mmcblk0p1 partition for Swap (type 0x82) and left the rest as /dev/mmcblk0p2 for Linux (type 0x83)
If you had a look at dmesg, you should see that your USB device has been detected by the kernel (normally as /dev/sda). Since we have our Slackware package on that device, this is what we are going to use for the source. I have found out however that:
- If you attempt to mount the directory manually and chose the Install from a pre-mounted directory option in SOURCE MEDIA SELECTION, MAKE SURE that your path ends with the slackware/ directory where the a/, ap/, d/... packages reside and NOT the one where extra/, isolinux/, kernels/... reside. If you don't, setup will happily list the packages but fail to install them and produce a completely irrelevant error!
- If you select "Install from a hard drive partition" but haven't previously mounted the partition, it will fail to mount the directory properly.
For now, we will use the "Install from a hard drive partition" option, but we first need to mount that partition manually.
And remember that the Slackware setup process uses /mnt for its own purpose, so you should not use that location as a mountpoint. So:
```
mkdir /usb

mount /dev/sda1 /usb

echo "mmc0" > /sys/class/leds/plug\:green\:health/trigger

setup
```
The "echo mmc0" line in there is a trick to get the blue LED act as an MMC card access indicator, as, if you're like me, you want to see some indication of r/w access to your root device.
Setup should be smart enough to detect your Swap & Linux mmc partitions, and format them (the default ext4 is a good choice for a filesystem). With regards to the SOURCE MEDIA SELECTION screen, you should now select "2 Install from a hard drive partition" then enter "/dev/sda1" as the device and "/armedslack" as the path (or whatever you called your directory on the USB). If you did mount your partition beforehand, this will work and you will then be able to select the packages straight off the USB.
From there, it's your run-of-the-mill package selection and installation. On my 8 GB SD Card, I found that selecting everything but GNU Emacs, TeX, KDEI and the Kernel sources resulted in a 3.7 GB installation, thus leaving about 2.9 GB free space, which is fine. Of course, if you want more space, you can fine tune your package selection or use a bigger SD card.

After you've gone through the rest of the configuration (network, etc) you can reboot the system and get back to the Marvell>> U-Boot prompt. There we will want to test that the custom kernel we've been using to boot Ubuntu can also boot our Slackware installation. Again, this supposes that you followed the steps from my earlier post regarding custom kernel installation and flashing. If not, now is probably a good time to follow this post to to flash the uImage-kirkwood kernel that you used for setup.

To test that your current NAND kernel can boot Slackware:

# Preliminary check to see if your bootargs_console variable is

# properly defined. if not, make sure you set it up

Marvell>> printenv bootargs_console

bootargs_console=console=ttyS0,115200

# This defines our root device. Make sure that use the right device, and that you mount it "ro"

Marvell>> setenv bootargs_root root=/dev/mmcblk0p2 rootfs=ext4 ro

Marvell>> setenv bootargs $(bootargs_console) $(bootargs_root)

Marvell>> printenv bootcmd

bootcmd=nand read.e 0x800000 0x100000 0x400000; bootm 0x800000

# Your bootcmd should only read the kernel and boot from it.

# If not, just issue the 2 commands above instead of what follows

Marvell>> run bootcmd

You should then be able to boot into your newly installed Slackware Linux. If not, you might want to check your parameters and install & flash a kernel that works.
Now, while we have confirmed that your system can indeed boot, we haven't yet saved anything on the flash to make it the default, so you need to reboot into the Marvell prompt once more, and this time:

# First, we start by saving the current working bootargs so that if you ever want

# to get back to Ubuntu, you can issue a "setenv bootargs $(bootargs_rescue)"

Marvell>> setenv bootargs_rescue $(bootargs)

Marvell>> setenv bootargs_root root=/dev/mmcblk0p2 rootfs=ext4 rootwait ro

Marvell>> setenv bootargs $(bootargs_console) $(bootargs_root)

Marvell>> saveenv

Saving Environment to NAND...

Erasing Nand...Writing to Nand... done

Marvell>> reset

After this last command, the plug should automatically boot into Slackware and you can finalize your installation.

IMPORTANT: You VERY MUCH want to add a rootwait parameter to your bootargs, which ensures that the SD is ready before the kernel attempts to mount it as rootfs. Otherwise, you may end up with maddening kernel panics when mounting rootfs, especially after a power failure, that are really due to the SD card not being ready yet.
Another alternative would be to add rootdelay=## but if you read here, you'll see that rootwait is the smarter option.

One thing you might want to do, as we did during the install, is set the blue LED as an MMC access indicator.
On my system, I did that by editing /etc/rc.d/rc.S and adding the line:
```
# Set the blue LED as SD access indicator

echo "mmc0" > /sys/class/leds/plug\:green\:health/trigger
```
just before the section
```
# Enable swapping:

/sbin/swapon -a
```

Using the SheevaPlug as an iSCSI target for Vista or Windows 7

2009-10-09T13:08:00.015+01:00

(Updated 2010.08.08)

Well, since I played with this iScsi stuff just for the sake of it, I might as well post a quick guide.

For this setup, I will be using the SheevaPlug running Slackware 13.0 (armedslack) on an SDIO card, a 100 GB HDD connected to the plug through USB (/dev/sda, but you could use a partition or even an image file) and a version of Windows that supports iSCSI (Vista or later). The SheevaPlug server has been creatively named "sheeva".

Compiling

The de-facto iSCSI target server tool for Linux is the Linux SCSI target framework (tgt) project (NB: you know you need to come up with better designation when your URL is actually shorter than your project name), so off we go and download the latest version (Important note: at the time of this post update, version 1.0.7 doesn't appear to work for the SheevaPlug, but 1.0.5 does).
We're pretty much going to follow the "./doc/README.iscsi" from the archive, so you might want to keep that file open for reference.
Once extracted on the SheevaPlug, cd to the usr/ directory and issue a "make ISCSI=1; make install". This package is not that big, so we can compile it on the Plug directly. Remember to add ISCSI=1, as default make will not have iSCSI.

Starting the target server daemon

After that, as root, you need to launch the daemon with

tgtd

The syslog should tell you that everything's rolling with a line like

Oct  9 12:46:23 sheeva tgtd: Target daemon logger with pid=1897 started!

Setting up the iSCSI target

This is where the bulk of our configuration occurs. What we're going to set below is an iSCSI drive named "sheeva:hd100" that maps to our /dev/sda HDD.

First we need to create that device:

root@sheeva:~# tgtadm --lld iscsi --mode target --op new --tid 1 --target sheeva:hd100
root@sheeva:~# tgtadm --lld iscsi --mode target --op show
Target 1: sheeva:hd100
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
    Type: controller
    SCSI ID: IET     00010000
    SCSI SN: beaf10
    Size: 0 MB
    Online: Yes
    Removable media: No
    Backing store type: rdwr
    Backing store path: None
Account information:
ACL information:
root@sheeva:~#

Now, despite the fact that a device with LUN0 was created as shown above, this is just a virtual controller device, not a physical target, and we still need to create a LUN for our device:

root@sheeva:~# tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 --backing-store /dev/sda
root@sheeva:~# tgtadm --lld iscsi --mode target --op show
Target 1: sheeva:hd100
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
    Type: controller
    SCSI ID: IET     00010000
    SCSI SN: beaf10
    Size: 0 MB
    Online: Yes
    Removable media: No
    Backing store type: rdwr
    Backing store path: None
LUN: 1
    Type: disk
    SCSI ID: IET     00010001
    SCSI SN: beaf11
    Size: 100030 MB
    Online: Yes
    Removable media: No
    Backing store type: rdwr
    Backing store path: /dev/sda
Account information:
ACL information:
root@sheeva:~#

This time, we have our physical device listed. Note that you could also mount a partition instead of a whole device, and make it appear as a device through iSCSI

At this stage, we have a target, but it doesn't accept inbound connections ("ACL information" is empty), so:

root@sheeva:~# tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address ALL
root@sheeva:~# tgtadm --lld iscsi --mode target --op show
Target 1: sheeva:hd100
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
    Type: controller
    SCSI ID: IET     00010000
    SCSI SN: beaf10
    Size: 0 MB
    Online: Yes
    Removable media: No
    Backing store type: rdwr
    Backing store path: None
LUN: 1
    Type: disk
    SCSI ID: IET     00010001
    SCSI SN: beaf11
    Size: 100030 MB
    Online: Yes
    Removable media: No
    Backing store type: rdwr
    Backing store path: /dev/sda
Account information:
ACL information:
ALL
root@sheeva:~#

Here you might want to find out if Windows is able to see our target, so you can fire up the iSCSI Initiator (in Control Panel -> Administrative Tools). If you are prompted to enable the iSCSI service on startup and modify the firewall rules, accept it.
Now, if you go to the "Targets" tab and press Refresh, you'll probably see that your target is not yet detected.
To make it appear in the target list, you want to go to the "Discovery" tab and click "Add/Discover Portal". Then provide the IP/DNS of your server. Port 3260 should be fine as it's used by the Target daemon on the Linux server.
After doing this, if you go to the "Targets" tab again, you should see something like this listed:
```
sheeva:hd100              Inactive
```
If not, you might want to check your daemon, firewall, etc. as you should be able to detect the target at this stage.
If you're fine with anybody being able to connect to our iSCSI target on your network, then you can just try to press "Log On" in the Targets tab, and you should be able to connect. You can then go to Computer Management, and you should be able to see the disk!
You should also see who is connect to your target in the "I_T nexus information" section from tgtadm in Linux:
```
root@sheeva:~# tgtadm --lld iscsi --mode target --op show
Target 1: sheeva:hd100
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 4
    Initiator: iqn.1991-05.com.microsoft:dusk
    Connection: 1
        IP Address: 192.168.1.184
LUN information:
LUN: 0
    Type: controller
    SCSI ID: IET     00010000
    SCSI SN: beaf10
    Size: 0 MB
    Online: Yes
    Removable media: No
    Backing store type: rdwr
    Backing store path: None
LUN: 1
    Type: disk
    SCSI ID: IET     00010001
    SCSI SN: beaf11
    Size: 100030 MB
    Online: Yes
    Removable media: No
    Backing store type: rdwr
    Backing store path: /dev/sda
Account information:
ACL information:
ALL
root@sheeva:~#
```
Pretty cool, heh?
If you do want to restrict access, you should of course follow the README.iscsi and add iSCSI target accounts. Then you can provide these credentials to the CHAP logon parameters in Windows (which can be accessed by clicking "Advanced" in the "Log On to Target" dialog. Be warned however that Windows will return a "CHAP secret given does not conform to the standard" error if your password is too short, so you need to make it at least 10 characters or more.
To disconnect from a target, on Vista, simply click on "Details", select its Identifier and click "Log off...". For Windows 7, Microsoft did the wise thing and added a "Disconnect" button.

IMPORTANT NOTE: Because we used LUN 1, and not LUN 0, if you attempt to boot from this newly created iSCSI disk, you MUST fill in the LUN parameter in the iSCSI root path syntax (the sequence of 4 columns that most guides will leave blank). In our case, if we wanted to use the device above for booting with gpxe/dnsmasq, we would add the following in dnsmasq.conf:
```
dhcp-option=net:gpxe,17,"iscsi:192.168.1.3:::1:sheeva:hd1"
```
Notice that there is a "1" for the LUN before "sheeva:hd1" and that we used :::1: instead of ::::. Likewise if you use a nonstandard port (different than 3260), you would use something like ::<custom port>:<LUN>:<target name>. This is detailed in RFC 4173, or you can find more information here.

Checking removable media for badblocks on Linux

2009-10-07T19:09:00.005+01:00

This comes handy when you are flashing firmwares from USB and don't want to take any risk with data corruptions (most firmwares images have checksums or CRCs, but who knows...)

Linux provides a badblock command to do just that. Obviously, this is a DESTRUCTIVE test (all data on your media will be erased without warning), and you'll need to reformat your media afterwards. Oh, and of course the badblocks command is available on the SheevaPlug.

badblocks -t random -svw <your_device>

Flashing a new kernel on the SheevaPlug

2009-10-07T11:04:00.008+01:00

This is the last piece of the puzzle with regards to our using a customized Ubuntu on the internal Flash. After that, we'll switch to using ArmedSlack on the MMC (but we'll keep that rescue Ubuntu image on the internal Flash).
Because we've pretty much done that before, flashing the new kernel is very straightforward, so I'm going to comment a little bit about the U-Boot commands we use.

As you'd expect, the "fatload usb 0 0x00800000 uImage" just takes the uImage file from the USB and loads it in RAM @ address 0x00800000, and it's this RAM image that we are going to flash. Because of the

mtdparts=orion_nand:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs)

option we pass to our U-Boot boot command (see what the command "printenv bootargs" reports in the Marvell>> prompt, or refer to the setenv we did previously), U-Boot expects the kernel uImage to be 0x400000 bytes in length (that's 4 MB) starting at address 0x100000 on the NAND Flash. Therefore, we will need to write the 4MB from our RAM containing the loaded uImage to 0x100000 on the Flash.
But before we can write the Flash we need to erase it.
Oh, and don't worry about our kernel image not being exactly 4 MB in length. As long as its size is less that 4 MB, you'll be fine.
If it's more though, then you will need to move the rootfs starting at 0x100000 + 4MB = 0x500000, and change your bootargs parameters accordingly.

All in all, to replace the existing kernel with a new

Marvell>> usb start
(Re)start USB...
USB:   scanning bus for devices... 2 USB Device(s) found
   scanning bus for storage devices... 1 Storage Device(s) found
Marvell>> fatload usb 0 0x00800000 uImage
reading uImage
...................................................................................................................................................................................................................................................................................................................

3152056 bytes read
Marvell>> nand erase clean 0x00100000 0x00400000

NAND erase: device 0 offset 0x100000, size 0x400000
Erasing at 0x4e0000 -- 100% complete. Cleanmarker written at 0x4e0000.
OK
Marvell>> nand write.e 0x00800000 0x00100000 0x00400000

NAND write: device 0 offset 0x100000, size 0x400000

Writing data at 0x4ff800 -- 100% complete.
4194304 bytes written: OK
Marvell>> reset

The bootup sequence shouldn't be that much different from the one you got when testing the new kernel in RAM

Using the blue LED on the SheevaPlug

2009-10-06T21:29:00.010+01:00

As mentioned here, it is possible to control the blue LED on the Plug and use it for notification.
The LED itself is accessible as /sys/class/leds/plug\:green\:health, and as mentioned in the Linux LED Documentation, it is possible to find out the available LED functions (triggers) through:

~# cat /sys/class/leds/plug\:green\:health/trigger
none nand-disk mmc0 timer heartbeat [default-on]

Thus, if you want a heartbeat effect, you would issue the command:

echo "heartbeat" > /sys/class/leds/plug\:green\:health/trigger

Now, on the more useful side, with a default Linux image running on the NAND flash, you might want to enable "nand-disk" to monitor said flash activity. Please be aware however that the LED behaviour in this case is quite different from your run-of-the-mill HDD access activity (which is a good thing - the less actual access to the NAND, the less wear and tear). In short, UBIFS does its job and reduces NAND accesses to the bare minimum, which in turn results in little NAND activity actually going on. A couple of well placed "sync" should suffice to convince you that "nand-disk" works as expected though.

If you're using the default Ubuntu running on the NAND Flash, you probably want to add the following to your /etc/rc.local then:

echo "nand-disk" > /sys/class/leds/plug\:green\:health/trigger

Recompiling a new kernel for the SheevaPlug

2009-10-06T10:54:00.033+01:00

There are many reasons why you would want to use a custom kernel on the plug. As far as I am concerned, the lack of ext4 and ntfs support in the kernel provided with the otherwise excellent SheevaPlug installer 1.0 system make a clear case for a custom kernel. After all, you might wanna mount external HDDs that are NTFS formatted, or, more important, you might want to install/access a Linux system running with ext4 (which I hear is nicer on flash based media devices than ext3), as we will do when we install armedslack on an SDIO card.

Setting up a cross-compiler for the SheevaPlug on Linux x86_64

While it is definitely possible to recompile a kernel on the SheevaPlug itself, let us just consider the raw compilation times observed when compiling a 2.6.31.2 kernel (exact same .config) on the Sheeva and on a more up to date PC:
- Quad Core PC (Slackware 13.0 x86_64 w/ ext4 fs on a RAID5 array) - 3:03 mins (183 secs)
- SheevaPlug (Ubuntu 9.04 w/ ext4 fs on a high speed SDIO card) - 57:42 mins (3462 secs)

That's one full order of magnitude right there (or about 20x slower)! And if you need a further case for cross-compilation, remember that extracting the kernel source takes ALOT of space. You will run out of space if you try to extract it on the internal flash NAND - it's just too big for what's left of the 512 MB space, and even on external media, you might want to think twice about using the limited space you have there.

If you're familiar with this blog, you know that there's no way we'll want to use any of the old, likely outdated, tools that came with the SheevaPlug SDK CD or internet binaries got compiled by people we don't know (and therefore don't trust!). Besides, the Cross compiler tgz I got on the CD has a big fat CRC error to start with, and it's 32 bit while our cross compiling rig is running Linux x86_64. If what we've seen with the 32 bit openocd binaries is any indication of what's in store for 64 bit users, we might as well cut to the chase and ignore anything that's not 64 bit.
What we do want, and always will want, is the most cutting edge and up to date technology we can lay our hands on. Therefore, we will of course recompile our own cross compiler for the SheevaPlug using the latest gcc, glibc and binutils.

Now, we could of course dive in straight ahead, and recompile, by hand, the whole cross-compiling shebang, but let's face it: your time is precious, so is mine, and if you just dive in head first in trying to produce a cross compiler from the GNU vanilla tarballs, whilst having no previous experience in the matter, you'll find soon enough that:

Compiling a cross compiler manually is absolute hell!
glibc and gcc have to be the worst offenders for breaking things that worked perfectly fine before with newer versions. All those rumours you've heard about needing to apply a massive amount of patches to gcc and gblic before you can compile them as cross-compilers? They're true!

Fact of the matter is I wasted more than one afternoon yesterday on the "Heck, I'm smarter than that - I'll go manual!" approach, and, while I now have a clearer idea of how one might get a cross-compiler actually compiled from scratch, I'll just say this: Do yourself a favour - if you value your time, don't waste it in trying to create a cross-compiling toolchain by hand.
Now, if you really, really, insist on compiling a whole toolchain manually, then my advice is to try to stick as close as possible to this guide (trust me!), but don't come whining that I didn't warn you you were about to step through the gates of hell beforehand...

Using CrossTool-NG

The MUCH better approach, to automate the cross compilation setup process is to use Yann E. Morin's most excellent Crosstool-NG. And, yes, despite being an automated tool, it will not compromise on our using the latest versions of gcc, glibc, binutils, etc.

So off we go:

cd /usr/src
mkdir sheeva
cd sheeva
wget http://ymorin.is-a-geek.org/download/crosstool-ng/crosstool-ng-1.4.2.tar.bz2
tar -xjvf crosstool-ng-1.4.2.tar.bz2
cd crosstool-ng-1.4.2
./configure
make
make install
cd ..
ct-ng menuconfig

Well, isn't that nice - a kernel like configuration menu. Alrighty then, here are the options you want to change for the SheevaPlug:

Paths and misc options

Try features marked as EXPERIMENTAL: check
Prefix directory: "/usr/local/${CT_TARGET}" (this will result in all the tools residing in /usr/local/arm-unknown-linux-gnueabi) or, preferred: "/usr/local/sheeva". One important note though: DO NOT USE /usr/local or any directory containing data you plan to keep. THIS DIRECTORY WILL BE ERASED. You have been warned.
Use LAN mirror: Let's try to use a mirror there, so - check

Prefer the mirror: check
Base URL: ftp://ftp.heanet.ie/mirrors/ftp.gnu.org/gnu/ (this is an Irish mirror - not entirely sure it works, as the downloads didn't seem much faster, but worth a try)

Stop after extracting tarballs: check (we'll need to patch glibc!)
Number of parallel jobs: 2x the number of cores you have. In my case that would be 8 (quad core)
Maximum log level to see: change from "INFO" to "EXTRA"

Target Options

Target Architecture: arm
Use EABI: check
Emit assembly for CPU: arm926ej-s
Floating point: change from "hardware (FPU)" to "software"

Toolchain options: don't change anything
Operating System:

Target OS: change from "bare-metal" to "linux"
Get kernel headers from: change from "kernel's 'headers_install'" to "Use custom headers"
Path to custom headers directory/tarball: "/usr/src/sheeva"

GMP and MPFR:

GMP and MPFR: check
GMP version: 4.2.4 (latest)
MPFR version: 2.4.1 (latest)

binutils

binutils version: 2.19.51.0.2

C compiler:

gcc version: 4.3.3
C++: check

C-library:

C library: glibc
glibc version: 2.9
Threading implementation to use: make sure "ntpl" is set.

The rest should be left untouched. For your reference, I am providing the Crosstool-NG .config file I used. Please note that if you use a recent version of glibc, you MUST have a threading implementation set, but you can NOT use linuxthreads, as it's been deprecated (you WILL get errors if you do select it), so the only option to use in nptl.

Kernel headers & glibc 2.9 patch

Before you can run the build, we still need to setup our kernel headers.
At the time of this writing, the most recent kernel is linux-2.6.31.2, so just download and extract it somewhere. Then, within the linux-2.6.31.2/ directory, run the command:

make headers_install ARCH=arm INSTALL_HDR_PATH=/usr/src/sheeva

The INSTALL_HDR_PATH should match exactly what you used for "Path to custom headers directory/tarball" in the menuconfig of Crosstool-NG.
You should now find that you have an "include" directory containing the relevant kernel headers in /usr/src/sheeva

Once you have everything setup, you need to go back to the sheeva directory (because this is where the ct-ng config resides) and launch the build with:

cd /usr/src/sheeva
ct-ng build

If you followed what we did above, you will note that we checked the "Stop after extracting tarballs" option, and indeed the build process stops after "Extracting and patching toolchain components". The reason we did that is because glibc 2.9 needs to be manually patched to prevent the following error (undefined reference to `_begin') from occurring:

[ALL  ]    /usr/src/sheeva/targets/arm-unknown-linux-gnueabi/build/build-libc/elf/librtld.os: In function `_dl_start_final':
[ALL  ]    raise.c:(.text+0x54c): undefined reference to `_begin'
[ALL  ]    /usr/src/sheeva/targets/arm-unknown-linux-gnueabi/build/gcc-core-shared/lib/gcc/arm-unknown-linux-gnueabi/4.3.3/../../../../arm-unknown-linux-gnueabi/bin/ld: /usr/src/sheeva/targets/arm-unknown-linux-gnueabi/build/build-libc/elf/ld.so: hidden symbol `_begin' isn't defined
[ALL  ]    /usr/src/sheeva/targets/arm-unknown-linux-gnueabi/build/gcc-core-shared/lib/gcc/arm-unknown-linux-gnueabi/4.3.3/../../../../arm-unknown-linux-gnueabi/bin/ld: final link failed: Nonrepresentable section on output
[ALL  ]    collect2: ld returned 1 exit status
(...)
[ERROR]    Build failed in step 'Installing C library'

This known problem is documented here and if you have a look at the diff file, you'll see that we can easily manually patch this problem by editing "targets/src/glibc-cvs-2.9/elf/Makefile" and replace, on line 314:

-e 's/\. = 0 + SIZEOF_HEADERS;/& _begin = . - SIZEOF_HEADERS;/' \

with

-e 's/\. = .* + SIZEOF_HEADERS;/& _begin = . - SIZEOF_HEADERS;/' \

Building the cross-compiler

Once you have made this change, just launch ct-ng menuconfig again (from the /usr/src/sheeva directory, as this is where the config file resides) and remove the checkmark on "Stop after extracting tarballs" (in the "Paths and misc options" menu)

Now, you can launch the actual build of the tools with

ct-ng build

This will take a little while (about 15 mins on my machine) but should complete successfully. If you have a multiple core system, hopefully you didn't forget to change the "Number of parallel jobs" option (which is equivalent to the -j option of make, i.e. number of concurrent jobs to run at once), as this dramatically reduces the time it takes to compile the toolchain. The -j option should always be your friend when compiling a large source, like a Linux kernel for instance.

Time for a tea or coffee break!

Note that, near the end of the compilation, you might get the error "[ERROR] libtool.m4: error: problem compiling FC test program". From what I gather however, FC is a test for the Fortran 90 compiler, which we couldn't care less about, so, unless you got another error, I will consider this build process of success, and you should now have the whole set of cross compiler tools in /usr/local/sheeva/bin. Time to compile ourselves a new kernel!

Installing U-boot mkimage

While we now have the tools to create arm binaries, we're still missing the last piece of the puzzle to boot a kernel, and that is the mkimage utility (which is called at the end of a successful kernel compilation), which turns a newly crafted kernel into something U-Boot can actually launch.

You guessed it, no way in hell we're gonna use the Marvell provided mkimage tool - we're gonna recompile our own from source of course!
And once more, the good guys at Debian (never underestimate a Debianer!) provide us with what we need:

cd /usr/src/sheeva
wget http://ftp.de.debian.org/debian/pool/main/u/uboot-mkimage/uboot-mkimage_0.4.tar.gz
tar -xzvf uboot-mkimage_0.4.tar.gz
make
make install

Easy as a pie!

Cross-Compiling a new kernel for the SheevaPlug

At long last, we're ready to compile us some custom kernel!
At this stage, we can more or less follow the Compiling Linux Kernel for the Plug Computer tutorial from openplug.org . Henceforth:

cd /usr/src/sheeva/linux-2.6.31.2
make ARCH=arm kirkwood_defconfig
make ARCH=arm menuconfig

There, change the options that you want. If you plan to keep using UBIFS (which you should have switched to during the SheevaPlug Installer 1.0 upgrade), then make sure that you enable BOTH:

Device Drivers ---> Memory Technology Device (MTD) support ---> UBI - Unsorted block images ---> Enable UBI
File systems ---> Miscellaneous filesystems ---> UBIFS file system support
Device Drivers ---> Generic Driver Options, and make sure that both "Create a kernel maintained /dev tmpfs (EXPERIMENTAL)" and "Automount devtmpfs at /dev" are enabled. Without those, you won't see init messages on the serial console after the root filesystem is mounted.

With this set, and the other default options from kirkwood_defconfig, you should be able to produce a usable kernel. Now is obviously a good time to add ext4 & NTFS suppot, and remove features (802.11 stack) that you do not want. Finally, we're ready to compile, and the kernel makes cross compilation very easy for us, as its CROSS_COMPILE feature enables us to specify exactly which tools it should use instead of the default ones. Thus, because we have an arm-unknown-linux-gnueabi-gcc, arm-unknown-linux-gnueabi-ld, etc. in /usr/local/sheeva/bin, we will use the line:

make -j8 ARCH=arm CROSS_COMPILE=/usr/local/sheeva/bin/arm-unknown-linux-gnueabi- uImage

and in about 3 minutes (vs. one hour if done on the SheevaPlug) you should end up with something like

Image Name:   Linux-2.6.31.2
Created:      Tue Oct  6 14:50:51 2009
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    3151992 Bytes = 3078.12 kB = 3.01 MB
Load Address: 0x00008000
Entry Point:  0x00008000
Image arch/arm/boot/uImage is ready

Building the modules

You probably still have a few modules to compile as well, which you can do as follows

make -j8 ARCH=arm CROSS_COMPILE=/usr/local/sheeva/bin/arm-unknown-linux-gnueabi- modules
make -j8 ARCH=arm CROSS_COMPILE=/usr/local/sheeva/bin/arm-unknown-linux-gnueabi- INSTALL_MOD_PATH=./modules modules_install

Just don't forget that you'll need to copy the content of the ./modules directory to your Ubuntu image to keep the kernel happy

Testing the new kernel on the plug

In a next post, we'll see how to actually flash that kernel on the plug. Before we do that though, we want to ensure that it can boot our current system, so we'll try to launch it from an USB stick.
Off we go then to copy your shiny new kernel to an USB stick with

mount /dev/sde1 /mnt/usb
cp arch/arm/boot/uImage /mnt/usb/
sync
umount /mnt/usb

For the test part, we'll need the serial console ("screen /dev/ttyUSB0 115200,-crtscts" or similar), with which we first reboot our system, and interrupt the autoboot to get to the Marvel >> prompt. From there, the following set of commands should boot the existing system with our new kernel:

usb start
fatload usb 0 0x00800000 uImage
bootm 0x00800000

Success? Then my work here is done.

Upgrading the factory SheevaPlug nand to a more efficient version

2009-10-04T12:38:00.013+01:00

If you've had a look at openplug.org, as you should have, you've read that the default installed NAND kind of blows (slow MMC/NAND accesses, old U-Boot version that lacks versatility to install newer kernels, etc.), so you'll want to upgrade the default NAND (u-boot and Ubuntu) to something that's more up to date.

Well, the good guys at openplug.org heard you and made a readily available package for ya (see SheevaPlug Installer in the HowTo's section), so off you go an download the big tarball.
Only thing is, well, they've customized their packages for the masses (that still run a 32 bit OS), not the classes (x64), and they insist on using a PHP installer (eeeeew!!!).
Are we gonna trust a one size-fits-all installer to cover our needs? Of course not!

The purpose of this guide then is to show you how to manually install the SheevaPlug Installer v1.0 from a Linux x86_64 environment, since, if you followed the previous post, you should now have a working openocd there. Of course, this stuff will work on 32 bit Linux as well.

For the purpose of this exercise, we'll assume that you have extracted the tar in /usr/src, and thus you have a /usr/src/sheevaplug-installer-v1.0/ directory and that you have a FAT32 formatted USB stick mounted in /mnt/usb

First we need to populate the USB stick.

cd /usr/src/sheevaplug-installer-v1.0/installer/
cp * /mnt/usb
sync

That's really all there is to it for the USB part. Once syncing is done, just plug the USB stick on the sheevaplug.

Then, log on the sheeva with the serial console, and reboot it. When you see the prompt "Hit any key to stop autoboot:" press a key. Now that the plug is in idle state, we'll start by re-flashing the U-Boot using openocd, which is definitely something you want to know how to do in case of trouble. Please note that, as shown below, you MUST launch openocd from the directory where your uboot.bin image is located. Also note that the flashing itself takes about 2-3 minutes, during which NOTHING will happen in the openocd console, so be patient.

cd uboot
openocd -f /usr/local/share/openocd/scripts/board/sheevaplug.cfg

Then from another Linux terminal

# telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
>  sheevaplug_init
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x000000d3 pc: 0xffff0000
MMU: disabled, D-Cache: disabled, I-Cache: disabled
0 0 1 0: 00052078
> sheevaplug_reflash_uboot
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x000000d3 pc: 0xffff0000
MMU: disabled, D-Cache: disabled, I-Cache: disabled
0 0 1 0: 00052078
NAND flash device 'NAND 512MiB 3,3V 8-bit' found
erased blocks 0 to 5 on NAND flash device #0 'NAND 512MiB 3,3V 8-bit'
[!!!! THE 2-3 MINUTES PAUSE HAPPENS HERE - BE PATIENT!!!!]
wrote file uboot.bin to NAND flash 0 up to offset 0x00074000 in 217.873932s
>

After that, the plug will reboot (into the new U-Boot), so make sure you have your serial console still running. The upgraded U-Boot should still let you boot the old factory installed Ubuntu Linux, so you might wanna check that everything's copacetic.

So far so good? Alright then, let's move to the next stage and flash that up to date Linux image.
This time we're not gonna use openocd but the flashing capabilities of the U-Boot, so you can "shutdown" (i.e. exit) openocd if not already done.

Once again, you need to reset the plug & interrupt the boot process to get to the Marvel>> prompt. There, we will install the new Ubuntu image by using the various commands from the U-Boot environment.
Three points that you need to know before starting this procedure:

Please note that it is necessary to boot the RAM image at the end to setup the NAND filesystem. Failure to do so will result in an unbootable Linux image, but this can be rectified by reloading the USB images and issuing the last command listed.
It is also a good idea, before you start this whole procedure, to spare a copy of your environment variables, which you'll get with the command 'printenv', as we're going to modify/create some of those. This way you should be able to revert back to your old environment if anything happens
The initial setup of the Linux image requires access to the USB stick, so make sure you leave it plugged in.

Marvell>> usb start
(Re)start USB...
USB:   scanning bus for devices... 2 USB Device(s) found
     scanning bus for storage devices... 1 Storage Device(s) found
Marvell>> fatload usb 0 0x00800000 uImage
reading uImage
...............................................................................................................................................................................................................................................................

2620504 bytes read
Marvell>> fatload usb 0 0x01100000 initrd
reading initrd
.....................................................................................................................................................................................................................................................................................................................................

3331626 bytes read
Marvell>> nand erase clean 0x00100000 0x00400000;

NAND erase: device 0 offset 0x100000, size 0x400000
Erasing at 0x4e0000 -- 100% complete. Cleanmarker written at 0x4e0000.
OK
Marvell>> nand write.e 0x00800000 0x00100000 0x00400000

NAND write: device 0 offset 0x100000, size 0x400000

Writing data at 0x4ff800 -- 100% complete.
 4194304 bytes written: OK
Marvell>> setenv mainlineLinux yes
Marvell>> setenv arcNumber 2097
Marvell>> setenv bootargs_root ubi.mtd=1 root=ubi0:rootfs rootfstype=ubifs
Marvell>> setenv mtdpartitions mtdparts=orion_nand:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs)
Marvell>> setenv bootargs_console console=ttyS0,115200
Marvell>> setenv bootargs $(bootargs_console) $(mtdpartitions) $(bootargs_root)
Marvell>> saveenv
Saving Environment to NAND...
Erasing Nand...Writing to Nand... done
Marvell>> setenv bootargs $(bootargs_console) $(mtdpartitions) root=/dev/ram0 rw ramdisk=0x01100000,8M install_type=nand; bootm 0x00800000 0x01100000

Now, because of the "bootm" command after the semicolon, this last command will make the plug boot automatically into the Linux copy in RAM, which needs to be done at least once to setup your initial system on the NAND. If all is well, this one time Linux installer will run & setup your system, restart the plug, and you'll get the new Ubuntu 9.04 booting which you'll find is A LOT faster than the factory one!

Next, we'll see how to recompile and install a vanilla kernel to boot the new distro.

OpenOCD and the SheevaPlug on Linux x86_64

2009-10-03T23:16:00.029+01:00

This is a tutorial for people like me who want to start with JTAG and OpenOCD access on the SheevaPlug. Most of this stuff regarding the installation of OpenOCD on Linux x86_64 is of course generic and can be used for other FTDI based JTAG devices like.

First of all, if you're running a 64 bit OS (Vista, Win7), a word of advice: Don't waste your time trying to get OpenOCD to work on a 64 bit Windows - there's no ready-made package that works (the 32 bit version just don't), so the only way is to recompile the whole shebang yourself, and it's a complete mess, as you will also need a 64 bit gcc-like compiler, and figure out how to get a 64 bit version of a the FTDI GPL library talk to the 64 bit libUSB drivers.
If I ever manage to get a Windows version going, you'll be the first to hear about it, but for the time being, if you value your time, stick with Linux.

Now, the distro I am using is Slackware x86_64. There are of course tutorials on how to compile OpenOCD for Linux here and there, but things are slightly different for 64 bit version, which is why I'm posting my own version.

Before anything else, you want to have the serial console to the SheevaPlug through the USB FTDI device. For that you need to compile the kernel driver for the "USB FTDI Single Port Serial Driver" either as a module, or directly in your kernel. You will find that option, in the kernel, under: "Device Drivers ---> USB support ---> USB Serial Converter support ---> USB FTDI Single Port Serial Driver"
Once you have the driver loaded, with the USB cable connected to the sheevaplug, you should get a /dev/ttyUSB0 device, which you can use as a regular serial port.
The best way to connect to the SheevaPlug in Linux (or to any serial terminal for that matter) has to be to use screen, with a command like:
```
screen /dev/ttyUSB0 115200,-crtscts
```
Once you have confirmed that you can connect to the plug using the FTDI USB Serial converter driver, you can move to building OpenOCD, as it's a good indication that Linux should have no trouble accessing the rest of the FTDI chip functionalities

OPTION 1: Using the proprietary libftd2xx library from FTDI, provided for completion. Note that using this option will kill any opened serial console whenever you launch openocd, whereas this does not occur with OPTION 2 below, which I strongly advise you to use.
On an x86_64 MUST pick up the 64 bit version of the Linux drivers from the FTDI website. At the time of this post, that means you must download libftd2xx0.4.16_x86_64.tar.gz, NOT libftd2xx0.4.16.tar.gz. What's more, when you install the library, it has to go into lib64, NOT lib else you will get the ominous
```
checking whether ftd2xx library works...
configure: error: Cannot build & run test program using ftd2xx.lib
```
Thus:
```
cd /usr/src
wget http://www.ftdichip.com/Drivers/D2XX/Linux/libftd2xx0.4.16_x86_64.tar.gz
tar -xzvf libftd2xx0.4.16_x86_64.tar.gz
cp libftd2xx0.4.16_x86_64/libftd2xx.so.0.4.16 /usr/local/lib64
ln -s /usr/local/lib64/libftd2xx.so.0.4.16 /usr/local/lib64/libftd2xx.so
# even if you specify --libdir=/usr/local/lib64 on openocd compilation
# it requires libftd2xx.so.0 in /usr/lib64!
ln -s /usr/local/lib64/libftd2xx.so.0.4.16 /usr/lib64/libftd2xx.so.0
```

OPTION 2 (PREFERRED): Using the GPL/Open source version of the FTDI library

cd /usr/src
wget http://www.intra2net.com/en/developer/libftdi/download/libftdi-0.16.tar.gz
tar -xzvf libftdi-0.16.tar.gz
cd libftdi-0.16
# by default, libftdi will go in /usr/local/lib, whereas, on x86_64, it should go to lib64, thus
./configure --with--libdir=/usr/local/lib64
make
make install
# openocd looks in /usr/lib64
ln -s /usr/local/lib64/libftdi.so.1.16.0 /usr/lib64/libftdi.so.1

Now it's time to get OpenOCD compiled. For that:

svn checkout svn://svn.berlios.de/openocd/trunk openocd
cd openocd
./bootstrap
# OPTION 1 - PROPRIETARY LIB FROM FTDI
./configure --libdir=/usr/local/lib64 --enable-maintainer-mode --enable-ft2232_ftd2xx
# OPTION 2 - GPL FTDI LIBRARY
./configure --libdir=/usr/local/lib64 --enable-maintainer-mode --enable-ft2232_libftdi
make
make install

Note that if you don't use the "--enable-maintainer-mode" option, you will get an error when make reaches the creation of the documentation, which is not a problem per se, but will prevent the openocd configuration scripts to be copied over on make install

If the above completed successfully, then you have openOCD ready to run. To connect to the SheevPlug then, issue a:

openocd -f /usr/local/share/openocd/scripts/board/sheevaplug.cfg

At this stage, if you are getting the following, then you need to reboot the plug and launch openOCD in the early stages of reboot:

Error: JTAG scan chain interrogation failed: all zeroes
Error: Check JTAG interface, timings, target power, etc.
Error: Trying to use configured scan chain anyway...
Error: feroceon.cpu: IR capture error; saw 0x00 not 0x..1
Warn : Errors during IR capture, continuing anyway...
Error: unexpected Feroceon EICE version signature

What you really want to see when launching openOCD is:

root@stella:/usr/src/openocd# openocd -f /usr/local/share/openocd/scripts/board/sheevaplug.cfg
Open On-Chip Debugger 0.3.0-in-development (2009-10-07-01:35) svn:2808
$URL: svn://svn.berlios.de/openocd/trunk/src/openocd.c $
For bug reports, read http://svn.berlios.de/svnroot/repos/openocd/trunk/BUGS
2000 kHz
jtag_nsrst_delay: 200
jtag_ntrst_delay: 200
dcc downloads are enabled
Warn : use 'feroceon.cpu' as target identifier, not '0'
Info : clock speed 2000 kHz
Info : JTAG tap: feroceon.cpu tap/device found: 0x20a023d3 (mfg: 0x1e9, part: 0x0a02, ver: 0x2)

OpenOCD is a client server software, so when you launch openOCD, you are really only launching the server part. To actually send JTAG commands and all that Jazz, you need to open a client session with:
```
telnet localhost 4444
```
This should then provide you with the On-Chip Debugger prompt.

Now, JTAG operations are a bit tricky (there are various JTAG states involved) so you can't really send any JTAG command you want (eg. NAND ID) any time you like and expect a meaningful reply. If you looked in the cfg file for the SheevaPlug, you'll see that before any NAND access operation is performed, a "sheevaplug_init" command is being called, so this is what we'll do. Once it's run, you will get a whole lot of interesting commands working, as illustrated with the session below:

> sheevaplug_init
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x000000d3 pc: 0xffff0000
MMU: disabled, D-Cache: disabled, I-Cache: disabled
0 0 1 0: 00052078
> scan_chain
     TapName            | Enabled |   IdCode      Expected    IrLen IrCap  IrMask Instr
---|--------------------|---------|------------|------------|------|------|------|---------
 0 | feroceon.cpu       |    Y    | 0x20a023d3 | 0x20a023d3 | 0x04 | 0x01 | 0x0f | 0x0c
> nand probe 0
NAND flash device 'NAND 512MiB 3,3V 8-bit' found
> nand list
#0: NAND 512MiB 3,3V 8-bit (Hynix) pagesize: 2048, buswidth: 8,
    blocksize: 131072, blocks: 4096
> nand info 0
#0: NAND 512MiB 3,3V 8-bit (Hynix) pagesize: 2048, buswidth: 8, erasesize: 131072
#0: 0x00000000 (128kB) erase state unknown (block condition unknown)
#1: 0x00020000 (128kB) erase state unknown (block condition unknown)
(...)
#4094: 0x1ffc0000 (128kB) erase state unknown (block condition unknown)
#4095: 0x1ffe0000 (128kB) erase state unknown (block condition unknown)

Accessing an SD card on the SheevaPlug

2009-10-02T18:34:00.003+01:00

Not something I knew beforehand.
USB devices are either auto detected or can be accessed as regular SCSI Disk devices (/dev/sda# - by the way, the SheevaPlug has no problem powering a 7200 rpm 2.5" HDD [UPDATE: until you try it for a few days, and its PSU blows up), but not SD Cards.

The device name to use then is: /dev/mmcblk0 for the raw device and /dev/mmcblk0p1 for the first partition.

Watching the Daily Show if you're not in the US

2009-10-02T17:45:00.005+01:00

Well, it finally happened. Daily video streaming for the last episode of the "The Daily Show" is no longer available from the Republic Of Ireland (was working fine 'till yesterday).

The solution:

Get a VPS, private server, whatever box in the US with enough bandwidth and an ssh server
Using putty add the following tunnel:
- source port: 80
- Destination: media.mtvnservices.com:80
Edit your C:\Windows\System32\drivers\etc\hosts file and add the line:
127.0.0.1 media.mtvnservices.com
Establish an ssh connection, and then open a browser to www.thedailyshow.com

"No soup for you"? Yeah, I'd like to see you try!

SheevaPlug - Let's get cracking!

2009-10-02T15:27:00.007+01:00

Alright, now that we have our new shiny SheevaPlug, let's get started.

Of course, first thing we want is a serial console to the Linux OS, so we'll plug the mini <-> std USB cable provided to one of our USB ports and find out if Vista (x64) is able to figure out the FT2232D driver itself... which it can't (surprise, surprise). No big deal. The PDF readme from the DevKit CD tells you that you can find the TeraTerm Windows drivers in the SheevaPlug_Host_SWsupportPackageWindowsHost.zip or, for a more up to date version, at http://www.ftdichip.com/Drivers/D2XX.htm. Installing the drivers with Vista autodetect is still a pain, as you don't get the option to browse to the driver directory (yes, Microsoft, all your users are complete tools - that's why they use your products in the first place!), but with the device manager, you're good to go. Yay, another COM port! Yay a set of USB Serial converters!
Alrighty, if we have our COM port, so we're good to go with the ever versatile putty. The FIRST thing you want to know is that the serial console is set to 115200 bauds, and the SECOND thing is that the default logon is root/nosoup4u (Globalscale: even their passwords are cool!). And we're in... Niiiiiice! dmesg, df -h, top, they're all waiting for you, but first you might as well wanna change the date & time, as mine was set to Apr 22 1953. That's like, way old! Also, since you're gonna wanna try a reboot to see what the console spills out, I'll just mention that there's a very loooong pause during boot after the line "eth0: link up, full duplex, speed 1 Gbps", so just be patient, you will get your logon prompt. Oh and you'll see warnings here and there as well, but what do you expect.
What do we have here? Ubuntu? Huh, well, at least it's Debian based and we have a good chunk of space left. We could have fared much much worse (read anything that's based on Red Hat). We'll install armedslack soon enough but let's get cracking then. Of course, we'll want to recompile stuff on the device itself (something that can't be done on a WRT!). Let's start by compiling the very handy memtester to do a little bit of torture test, and see how much the baby heats up when using RAM.
According to google, installing gcc on the SheevaPlug ios as easy as running: "apt-get install build-essential"... Except that you are missing the cache directory so the command fails. You need to issue a
```
mkdir -p /var/cache/apt/archives/partial
```
before you can use apt-get (and you might want to add this line to your /etc/rc.local as well). But we're still not out of the woods yet, as the package upgrade seems to start up fine, but after a while, you get errors like:
```
Failed to fetch http://ports.ubuntu.com/pool/main/b/binutils/binutils_2.19.0.20090110-0ubuntu1_armel.deb  404 Not Found
```
Checking that URL confirms that the package is no longer hosted there.
Drats!
The easiest solution at this stage is to update your system as a whole with "apt-get update". This might take a while, but it seemed to get the problem sorted, as the subsequent "apt-get install build-essential" completed hapilly. You might also want to run an "apt-get autoremove" after that, to free up some space. Good, now we can compile natively on the plug.
"-bash: wget: command not found"? Who on earth delivers a system without wget? "apt-get install wget" then, but sheesh! But then...
```
root@debian:/usr/src# wget http://pyropus.ca/software/memtester/old-versions/memtester-4.1.2.tar.gz
--2009-10-02 14:05:28--  http://pyropus.ca/software/memtester/old-versions/memtester-4.1.2.tar.gz
Resolving pyropus.ca... failed: Name or service not known.
wget: unable to resolve host address `pyropus.ca'
```
Indeed, DNS resolution doesn't seem to work out of the box and the default /etc/resolv.conf goes something like:
```
domain lan
search lan
nameserver 127.0.0.1
```
It's clear that with only these 3 lines, you're not gonna get very far (it's a wonder apt-get still managed to find its way - must be using direct IPs). Let's just comment the domain & search lines and add our ISP's nameservers before the 127.0.0.1 to get some resolution going. Now, please be aware that any changes you make here won't hold on reboot, so for more details on how to fix the default annoyance, you might as well have a look at the New Plugger Howto from openplug.org.
Good, now we can wget memtester, compile it and run it (and it looks like our RAM is good). Maybe I'll try mprime later on as well... But to complete the setup of a half decent Linux system, let's issue an "apt-get install screen.

Next, we'll try to play with JTAG and OpenOCD...

SheevaPlug - first impressions

2009-10-02T11:42:00.029+01:00

At last, my SheevaPlug Development Kit has arrived!

For those who don't know yet what the SheevaPlug is, you can have a look at this short CNET article or Marvell's slightly longer blurb about this wonderful little device.
I ordered mine (UK version) from Globalscale about 2-3 weeks ago, to basically replace my long trusted WRT54G running openWRT with something that has a little more punch and I/O capabilities. The truth of the matter is, I almost never use the WRT for WiFi, but having an always on & completely customizable Linux low power "server" has proved invaluable (Samba share for easy data interchange between any machine, DHCP server, bootp/tftp server, nmap & other networking tools provider, etc.), and I had been waiting for some time to see something like Marvell's plug computer come out, as it's basically a glorified hacked WRT54G, without the unneeded Wifi/network hub features.

For those who can't be bothered to read the whole post (and who can blame them!), I'll jump right into the main points of my first impressions:

The not so good:

Globascale's delays in shipment, and total lack of response when asked for updates. Granted delays are likely unavoidable (I doubt they're shipping Sheevaplugs by the truckload right now), but that doesn't explain why it took about 2 weeks of repeated e-mails to sales to get a simple status update on the order.
No choice but to use Fedex for shipping if you're not in the US. Granted, once it's shipped, the delivery is fast indeed, but considering the shipping delay, you might as well want to use regular post and spare a few bucks.
Also, since Fedex are really happy to help with the obnoxious EU VAT & Customs Duty extortion scheme, you will be hit by 46% retroactive VAT on import (€31.20 - I really wish I was making this up), meaning that, as usual, no matter what you do, amount in $ = amount in €, and you still have to pay €99 for a bloody $99 item. Better believe this is the last time I import anything with Fedex.
HEED MY ADVICE - DO NOT IMPORT ANYTHING WITH FEDEX, EVER!!!
Unlike the most excellent US removable plug (see below), the UK plug is just a common adapter to fit over the US socket, so it's not as sleekly designed. Not that big a deal, but it adds to the overall height of the device when used in plug mode. It would have been nice to see an integrated UK plug in the way the US one is. Also, the Globalscale UK devkit photos don't make that as clear as they could.
The mainboard is now rev. 1.3 and very different from the reference mainboard you see in the documentation (see below). For starters, there is no longer a separate mainboard for JTAG and SDIO, and the solder-able port for an ARM JTAG connector is gone. Gone as well is the generic GPIO connector. There still seems to be a small JTAG port, but with only 8 connectors and a non-standard smaller socket too be fitted in, it will be a pain to interface with. Not sure if many people would put single board vs 2 separate boards design as a disadvantage, but I was kind of planning on using the standalone expansion board as a cheap JTAG device (it has the very well known FT2232 JTAG IC on it) but now using the SheevaPlug hardware as a JTAG interface to other boards, or as a general purpose IO device, is going to be less of a possibility for your regular electronics hobbyist.
UPDATE: for some of the new ports pinouts (still no JTAG desc), have a look here
Even idle, the device seems to run a bit hotter than I anticipated (It's certainly running hotter than my WRT54G). Definitely not that hot (akin to one of the numerous power bricks you have in your house), and most likely the heat is due to the PSU rather than the chip themselves, but still, I've seen power bricks run cooler than that (the 12V DC WRT brick being one of those). Of course, anybody who hears "heat" automatically thinks "increased power consumption", so I don't know how that translates into wasted wattage. My guess is this might be due to using a 110V/220V compatible PSU. Using a localized source voltage PSU for 220V -> 5V delivery would probably reduce the heat.
[UPDATE 2009.10.08] The Plug does run scaringly hot when running at full load for a prolonged period of time, as I just found out through a rogue process.
The metal parts on both USB and network sockets are almost burning when running for a long time at 100% of the CPU, and you wouldn't want to leave something in contact with them then. Still probably not hot enough to light up paper or plastic that would be in contact, yet quite scary... And it looks like the PSU unit does play some part in raising the whole temperature of the plug on full load. I also found that I had to enlarge some of the vents on my plug with a knife because they had been pourly moulded and were not entirely open. Still, I don't think Even then, there are too few vents on the current Plug and I don't think it is designed for adequate cooling! Something that definitely needs some improvement for later models.
My advice: make sure you leave ample space around the plug, and check that all your vents are fully open.

The good:

Brilliant design from Globalscale. As somebody else said, whoever designed that removable socket for corded/plugged needs to have both a raise and a corner office. And despite the point I made above, the single mainboard layout is very well done (and rumour has it there is even an eSata port in there). Those guys have sharp engineers! The whole device itself feels both sturdy, inconspicuous, yet elegant. Even the cardboard packaging the plug is shipped in is very good design, so much so that I almost felt like taking pictures of the packaging. But then I thought, only Apple fanboys ever do something like that...
Does exactly what it says on the tin. The layout design might have changed, but you still get 2 USB ports (1 for debug), 1 SDIO port, 1 Gb Ethernet port and a computer with 512 MB RAM, 512 MB Flash, a most excellent 1.2 GHz ARM CPU, plus a JTAG device that's compatible with OpenOCD. Who could ask for more?
Can power a 2.5" USB HDD (tested with both a 5400 rpm 100 GB and a 7200 rpm 320 GB HDD). Impressive!
Loads of free space on the flash with the default Ubuntu installation

Alright, now it's time for opening the device and posting some pictures! Wait, are you telling me that the first thing you do when receiving a cool new toy is NOT tearing it apart to see what's inside? What on earth are you doing with your life?

Some pictures

That very well done removable socket/cord plug concealer

From SheevaPlug

Opening the plug (just remove the rubber pads to find the screws)

From SheevaPlug

Mainboard - top

From SheevaPlug

Mainboard - underside, with heatsink

From SheevaPlug

Mainboard - underside, without heatsink

From SheevaPlug

The device in action

From SheevaPlug

LILO Warning: The initial RAM disk is too big

2009-10-01T19:02:00.011+01:00

Just re-installed Slackware 13.0 x64 on the HTPC, this time using a small boot partition on the extra non fakeRAIDed HDD, which I recently added to the whole rig. This 4th HDD is now the boot disk for the whole rig, and contains the kernel and initrd (while the Linux partition itself is on the RAID5 set).

Install went without much of a hitch, and even the first few kernel recompilations we fine, but lately, after adding more kernel features, I got the following LILO warning and the kernel simply doesn't boot:

    Warning: The initial RAM disk is too big to fit between the kernel and
   the 15M-16M memory hole.  It will be loaded in the highest memory as
   though the configuration file specified "large-memory" and it will
   be assumed that the BIOS supports memory moves above 16M.

So much for checking the "Optimizing for size" kernel option.

The root of the matter seems to be that uncompressed kernel + uncompressed initrd must fit within the 0-15 MB low memory addresses, and it looks like a bzipped 4.5 MB kernel + a gzipped 1.4 MB initrd don't cut it, regardless of whether your BIOS has set a hole for the 15M-16M range.

Unlike what you might read elsewhere ("Remove options from your kernel to reduce its size! / Why are you using such a huge kernel?" - how's that helpful, guys?), there is a solution right there in the kernel conf, through the "General Setup -> Initramfs source file(s)" option.
Somehow when you do that, the kernel is able to find all the RAM it needs for the initrd.

Now, I'll grant you that this makes it less convenient to modify your initrd options on the fly, yet I believe this is better than being restricted in your choice of kernel features. Plus it appears that the kernel boots a little bit faster when using the emebedded initrd.

Of course, that means that you need to create your uncompressed cpio archive from your initrd-tree before you can point to it, which can easily be done as follows:

cd /boot/initrd-tree
find . | cpio -H newc -o > ../initrd.cpio

Grouping XML nodes using XSL 2.0

2009-09-18T01:01:00.005+01:00

Man, that was painful!!! Curse those XML Committee morons that should have designed a transformation language based on Perl, rather than come with the atrocity that is XSL!

So, let's say you have the following XML (This is a list of cross references between 8051 banks, custom generated by a popular disassembler), which you already sorted using the other XSL I posted today:

<banks>
 <B0_000E/>
 <B0_000E/>
 <B0_000E/>
 <B0_000E/>
 <B0_49F8/>
 <B0_49F8/>
 <B0_49F8/>
 <B0_49F8/>
 <B0_49F8/>
 <B0_49F8/>
 <B0_520A/>
 <B0_520A>
    <BD_2E9D/>
    <BD_C4B5/>
 </B0_520A>
 <B0_520A/>
 <B0_54F9/>
 <B0_54F9/>
 <B0_54F9>
    <BB_E851/>
 </B0_54F9>
 <B0_54F9/>
 <B0_59EC>
    <B0_012F/>
 </B0_59EC>
 <B0_59EC>
    <BE_012F/>
 </B0_59EC>
 <B0_59EC>
    <BF_012F/>
 </B0_59EC>
 <B0_CC5E/>
 <B0_CC5E>
    <BC_DAA6/>
    <BC_DF2E/>
 </B0_CC5E>
 <B0_CC5E>
    <BB_DA1C/>
 </B0_CC5E>
 <B0_CC5E/>
  <B0_E022/>
 <B0_E022>
    <BE_E439/>
    <BE_E458/>
    <BE_EE05/>
    <BE_EE28/>
    <BE_F15B/>
    <BE_F17A/>
    <BE_F8E5/>
    <BE_F8F9/>
 </B0_E022>
 <B0_E022>
    <BF_3BF0/>
    <BF_6C6B/>
    <BF_7F7E/>
    <BF_7F9D/>
    <BF_7FE0/>
    <BF_7FFF/>
    <BF_CD2C/>
    <BF_D103/>
    <BF_D126/>
    <BF_D146/>
    <BF_D169/>
 </B0_E022>
 <B0_E022/>
 <B0_E022>
    <BC_CE5A/>
 </B0_E022>
 <B0_E022/>
 <B0_E022/>
 <B0_E201/>
 <B0_E201/>
</banks>

Obviously now, you would like to move all the sub nodes into a single one and remove all the duplicates to obtain a file like this:

<banks>
 <B0_000E/>
 <B0_49F8/>
 <B0_520A>
    <BD_2E9D/>
    <BD_C4B5/>
 </B0_520A>
 <B0_54F9>
    <BB_E851/>
 </B0_54F9>
 <B0_59EC>
    <B0_012F/>
    <BE_012F/>
    <BF_012F/>
 </B0_59EC>
 <B0_CC5E>
    <BC_DAA6/>
    <BC_DF2E/>
    <BB_DA1C/>
 </B0_CC5E>
 <B0_E022>
    <BE_E439/>
    <BE_E458/>
    <BE_EE05/>
    <BE_EE28/>
    <BE_F15B/>
    <BE_F17A/>
    <BE_F8E5/>
    <BE_F8F9/>
    <BF_3BF0/>
    <BF_6C6B/>
    <BF_7F7E/>
    <BF_7F9D/>
    <BF_7FE0/>
    <BF_7FFF/>
    <BF_CD2C/>
    <BF_D103/>
    <BF_D126/>
    <BF_D146/>
    <BF_D169/>
    <BC_CE5A/>
 </B0_E022>
 <B0_E201/>
</banks>

Well, using the XSL (2.0) below, now you can!

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output omit-xml-declaration="yes" indent="yes"/>

<xsl:template match="/">
  <banks> <!-- Our XML root element. Modify according to yours! -->
    <!-- Select all nodes that are immediate children of our root
         element ("/*/*") and group them according to their node
         names ("local-name()")  -->
    <xsl:for-each-group select="/*/*" group-by="local-name()">
      <!-- Output a (single) node for each of these groups -->
      <xsl:element name="{local-name()}">
      <!-- NB: To remove empty nodes, move the <xsl:element>
           section inside the <xsl:for-each-group> below, and use
           {local-name(parent::*)} instead of {local-name()} -->
        <!-- Select all the children nodes from the above group
             ("current-group()/*) - Don't bother to group them. -->
        <xsl:for-each-group select="current-group()/*" group-by=".">
          <!-- Here we copy the whole (group) branch as is -->
          <xsl:copy-of select="current-group()"/>
        </xsl:for-each-group>
      </xsl:element>
    </xsl:for-each-group>
  </banks> <!-- Close our root -->
</xsl:template>
</xsl:stylesheet>

Now, Vista, where's my file actually?

2009-09-17T19:25:00.004+01:00

If security done backwards is better than no security at all, then Microsoft are the kings of running in reverse.

You might have used some programs to edit files in "System" directories, only to wonder why the files that are there don't reflect your changes.
The answer is that, when Microsoft is unhappy about the location you edit a file in (eg: C:\Program Files (x86)\<some program>\<some file>), it creates a duplicate in C:\Users\<your user name>\AppData\Local\VirtualStore\ (eg: C:\Users<your user name>\AppData\Local\VirtualStore\Program Files (x86)\<some program>\<some file>).

Pretty sure that having 2 copies of one file is gonna be a thrill for people who want to backup their data...

...and the class blurb for SyntaxHighlighter

2009-09-17T17:11:00.003+01:00

Great tool that SyntaxHighlighter. Followed this post and got exactly what's written on the tin.

Now, since I'm of a forgetful nature, the syntax for the <pre> modifier is <pre class="brush: <brushname>"> where <brushname> is one of text, bash, xml, cpp or any of the keywords highlighted here.

...and the UltraEdit Macro script to convert code for blogger publishing

2009-09-17T15:58:00.008+01:00

Courtesy of Rob, to which I added the & -> & conversion:

InsertMode
ColumnModeOff
HexOff
UnixReOff
IfSel
Find SelectText "&"
Replace All "&amp;"
Find SelectText "<"
Replace All "&lt;"
Find SelectText ">"
Replace All "&gt;"
Find SelectText """
Replace All "&quot;"
Find SelectText "^t"
Replace All "  "
Else
GotoLine 1 1
Find "&"
Replace All "&amp;"
Find "<"
Replace All "&lt;"
Find ">"
Replace All "&gt;"
Find """
Replace All "&quot;"
Find "^t"
Replace All "  "
EndIf

XSLT script to sort nodes alphabetically

2009-09-17T15:35:00.015+01:00

The simple XSL sample you have been looking for. Works great with Saxon. "version" set to 2.0 here to prevent a warning with the latest version of Saxon, but you can change it to 1.0 for older XSLT engines.

<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output omit-xml-declaration="no" indent="yes"/>
<xsl:strip-space elements="*" />

<xsl:template match="/">
  <xsl:apply-templates />
</xsl:template>

<xsl:template match="@*|node()">
  <xsl:copy>
    <xsl:apply-templates select="@*|node()">
      <xsl:sort select="@typeName"/>
      <xsl:sort select="name(.)"/>
      <xsl:sort />
    </xsl:apply-templates>
  </xsl:copy>
</xsl:template>

</xsl:stylesheet>

Oh, and the saxon commandline syntax to apply an XSL (which should really be provided in a straightforward fashion in the commandline help) is:

C:\>saxon -s:banks.xml -xsl:sort.xsl -o:banks_sorted.xml

.NET, or the solution to a problem that didn't exist

2009-09-16T15:06:00.004+01:00

So, you're using Visual Studio Express and you want to use that supposedly nice SerialPort .NET class (after spending more than half a day trying to make a standard COM port app that can talk nicely to putty - hint: don't rely on anything else but putty's actual port initialization code if you want something that works!).

And yeah, you're like me, so you don't care much about whatever the hell .NET brings to the OO paradigm. You just want to use a class that works ASAP, so you followed the tutorials on the net that tell you: "Just add a 'using namespace System::IO::Ports;' and then you can define a SerialPort instance", which you promptly did.

But now, of course, you're getting the bleeping error:

error C2039: 'Ports' : is not a member of 'System::IO'

Yeah, that's because with all this supposedly improved technology, Visual Studio is still not able to tell you that you need to add a

#include "system.dll"

before you can see ALL of the System namespace. "Intellisense", my ass!

In short: C0239 = need to add a #include to a dll (?!?, when did that become a C++ standard) BEFORE you can start using "using namespace"

Oh, and good luck figuring that one out if you don't (want to) have a clue about .NET...

More 8032 / 8051 trickery

2009-08-28T11:05:00.008+01:00

Don't worry, the (serial) printf handler is coming. In the meantime, some more hard to spot Mediatek 8032/8051 (Geez, couldn't at least agree on the base architecture name, and use declinations that make sense?) compiler trickery with lcall:

ROM:C32C                 lcall   ROM_B613
ROM:C32F                 lcall   copy_4_bytes_to_xRAM ; input: DPTR = dest xRAM address
ROM:C32F                                         ;        PC   = src bytes address
ROM:C32F ; ---------------------------------------------------------------------------
ROM:C332                 .byte 0x00,0xC,0x00,0x00         ; 0
ROM:C336 ; ---------------------------------------------------------------------------
ROM:C336                 mov     DPTR, #0xFFFC
ROM:C339                 lcall   ROM_B613
ROM:C33C                 lcall   copy_4_bytes_to_xRAM ; input: DPTR = dest xRAM address
ROM:C33C                                         ;        PC   = src bytes address
ROM:C33C ; ---------------------------------------------------------------------------
ROM:C33F                 .byte 0,0xC,0,0         ; 0
ROM:C343 ; ---------------------------------------------------------------------------
ROM:C343                 mov     R6, #0x12
ROM:C345                 mov     R7, #0

When people extensively start/have to use instructions in other ways than they were designed for (shouldn't all lcalls' return flow of execution pcik up right after the instruction itself?), you know that you have a poorly designed CPU, no matter what people manage to achieve with it.

Yeah, it's another lcall'ed function that doesn't return where it's supposed to and breaks our nice disassembly flow. Thanks for making us having to manually edit our subroutine ends in IDA Pro, morons!

So, how does it work this time? Similarly to the previous trick actually:

ROM:B498 ; =============== S U B R O U T I N E =======================================
ROM:B498
ROM:B498 ; input: DPTR = dest xRAM address
ROM:B498 ;        PC   = src bytes address
ROM:B498
ROM:B498 copy_4_bytes_to_xRAM:                   ; CODE XREF: ROM:2A49p
ROM:B498                                         ; ROM:2A77p ...
ROM:B498                 mov     R0, DPL
ROM:B49A                 mov     B, DPH          ; DPTRd [DPHd DPLd] -> B R0 = Dest Pointer
ROM:B49D                 pop     DPH
ROM:B49F                 pop     DPL             ; DPTRs [DPHs DPLs] = Src Pointer
ROM:B4A1                 lcall   copy_byte
ROM:B4A4                 lcall   copy_byte
ROM:B4A7                 lcall   copy_byte
ROM:B4AA                 lcall   copy_byte
ROM:B4AD                 clr     A
ROM:B4AE                 jmp     @A+DPTR         ; DPTRs (= original return PC) has
ROM:B4AE ; End of function copy_4_bytes_to_xRAM  ; been incremented 4 times at this stage
ROM:B4AE
ROM:B4AF
ROM:B4AF ; =============== S U B R O U T I N E =======================================
ROM:B4AF
ROM:B4AF
ROM:B4AF copy_byte:                              ; CODE XREF: copy_4_bytes_to_xRAM+9p
ROM:B4AF                                         ; copy_4_bytes_to_xRAM+Cp ...
ROM:B4AF                 clr     A
ROM:B4B0                 movc    A, @A+DPTR      ; read [DPTRs]
ROM:B4B1                 inc     DPTR            ; DPTRs++ (as well as [DPTRs] -> A)
ROM:B4B2                 xch     A, DPH
ROM:B4B4                 xch     A, B            ; DPTRs <-> DPTRd (B R0)
ROM:B4B6                 xch     A, DPH
ROM:B4B8                 xch     A, R0
ROM:B4B9                 xch     A, DPL
ROM:B4BB                 xch     A, R0           ; A is still [DPTRs] at this stage
ROM:B4BC                 movx    @DPTR, A        ; [DPTRs] -> [DPTRd]
ROM:B4BD                 inc     DPTR            ; DPTRd++
ROM:B4BE                 xch     A, DPH          ; DPTRs <-> DPTRd
ROM:B4C0                 xch     A, B
ROM:B4C2                 xch     A, DPH
ROM:B4C4                 xch     A, R0
ROM:B4C5                 xch     A, DPL
ROM:B4C7                 xch     A, R0
ROM:B4C8                 ret
ROM:B4C8 ; End of function copy_byte

Gotta love these scores of xch instructions. Kind of the 3 cups & one red ball street magician classic. The only thing you need to know though, is that all a sequence like the following 3 lines does:

xch A, register1
xch A, register2
xch A, register1

is simply exchanging the content of register 1 & register 2, and keeping A unchanged.

I won't say this isn't a nice trick to access a small amount of data right within your program, but I'll take my general purpose register generous 68000 any time over your crappy 8051. It's a wonder really: How comes there are so few consumer devices using embedded 68000, when it would be so much more efficient for embedded applications? If you're gonna play with these kind of intel x32/x51 hurdles, it won't be that much different to go RISC all the way!

Search for low pointer values in IDA Pro

2009-08-26T16:59:00.004+01:00

Use the following regexp:

DPTR, #0x[0-9A-F]{3}$

This will lock only on DPTR assignment values in 0x0000 0x0FFF

Switch statements, 8032/8051 style

2009-08-26T12:55:00.005+01:00

Now this next trick is a little bit more clever.

If, when disassembling 8032 / 8051 code, you ever see the start of an lcall'ed routine that looks like this

ROM:0000B5BA                 pop     DPH             ; Data Pointer, High Byte
ROM:0000B5BC                 pop     DPL             ; Data Pointer, Low Byte

and then proceeds to use DPTR for movc instructions, the effective DPTR address used in that routine will be the next address after the LCALL (because lcall did push the current PC on the stack)

This is effectively used by, hum, some code, to implement switch statements that might be a bit tricky to detect during reverse engineering, as the disassembler obviously expect lcall to return, and will start disassembly the switch table.

Example code:

ROM:00006CA7 ROM_6CA7:                               ; CODE XREF: ROM_6C60+15j
ROM:00006CA7                                         ; ROM_6C60+1Bj ...
ROM:00006CA7                 mov     DPTR, #0xC46A
ROM:00006CAA                 movx    A, @DPTR
ROM:00006CAB                 xrl     A, #0x60
ROM:00006CAD                 jnz     ROM_6CF4
ROM:00006CAF                 mov     DPTR, #0xB03E
ROM:00006CB2                 movx    A, @DPTR
ROM:00006CB3                 lcall   case_switch_byte
ROM:00006CB3 ; ---------------------------------------------------------------------------
ROM:00006CB6                 .word ROM_6CE4
ROM:00006CB8                 .byte 0
ROM:00006CB9                 .word ROM_6CE4
ROM:00006CBB                 .byte 0x20
ROM:00006CBC                 .word ROM_6CE0
ROM:00006CBE                 .byte 0x2A
ROM:00006CBF                 .word ROM_6CDC
ROM:00006CC1                 .byte 0x2B
ROM:00006CC2                 .word ROM_6CDE
ROM:00006CC4                 .byte 0x2D
ROM:00006CC5                 .word ROM_6CE2
ROM:00006CC7                 .byte 0x2F
ROM:00006CC8                 .word ROM_6CDA
ROM:00006CCA                 .byte 0x39
ROM:00006CCB                 .word ROM_6CD2
ROM:00006CCD                 .byte 0x5A
ROM:00006CCE                 .word 0
ROM:00006CD0                 .word 0x6CEC
ROM:00006CD2 ; ---------------------------------------------------------------------------
ROM:00006CD2
ROM:00006CD2 ROM_6CD2:                               ; DATA XREF: ROM_6C60+6Bo
ROM:00006CD2                 mov     DPTR, #0xB03E
ROM:00006CD5                 mov     A, #0x30 ; '0'
ROM:00006CD7                 movx    @DPTR, A
ROM:00006CD8                 sjmp    ROM_6D54
ROM:00006CDA ; ---------------------------------------------------------------------------

with case_switch_byte being:

ROM:B5BA
ROM:B5BA ; =============== S U B R O U T I N E =======================================
ROM:B5BA
ROM:B5BA ; Iput: A = matching case value (byte)
ROM:B5BA
ROM:B5BA case_switch_byte:                       ; CODE XREF: ROM:4046p
ROM:B5BA                                         ; ROM_4552+178p ...
ROM:B5BA                 pop     DPH             ; Data Pointer, High Byte
ROM:B5BC                 pop     DPL             ; DPTR = lcall return address
ROM:B5BE                 mov     R0, A
ROM:B5BF
ROM:B5BF loop_through_cases:                     ; CODE XREF: case_switch_byte+24j
ROM:B5BF                 clr     A
ROM:B5C0                 movc    A, @A+DPTR      ; NB: @ is confusing. It's just A+DPTR
ROM:B5C1                 jnz     valid_dest      ; make sure dest @ != 0
ROM:B5C3                 mov     A, #1
ROM:B5C5                 movc    A, @A+DPTR
ROM:B5C6                 jnz     valid_dest
ROM:B5C8                 inc     DPTR
ROM:B5C9                 inc     DPTR            ; if null dest, just use the next
ROM:B5C9                                         ; word as dest address
ROM:B5CA
ROM:B5CA dest_match:                             ; CODE XREF: case_switch_byte+1Fj
ROM:B5CA                 movc    A, @A+DPTR
ROM:B5CB                 mov     R0, A
ROM:B5CC                 mov     A, #1
ROM:B5CE                 movc    A, @A+DPTR
ROM:B5CF                 mov     DPL, A          ; Data Pointer, Low Byte
ROM:B5D1                 mov     DPH, R0         ; dest into DPTR
ROM:B5D3                 clr     A
ROM:B5D4                 jmp     @A+DPTR
ROM:B5D5 ; ---------------------------------------------------------------------------
ROM:B5D5
ROM:B5D5 valid_dest:                             ; CODE XREF: case_switch_byte+7j
ROM:B5D5                                         ; case_switch_byte+Cj
ROM:B5D5                 mov     A, #2
ROM:B5D7                 movc    A, @A+DPTR
ROM:B5D8                 xrl     A, R0           ; cmp val with parameter
ROM:B5D9                 jz      dest_match
ROM:B5DB                 inc     DPTR
ROM:B5DC                 inc     DPTR
ROM:B5DD                 inc     DPTR            ; skip 3 bytes to next switch table entry
ROM:B5DE                 sjmp    loop_through_cases
ROM:B5DE ; End of function case_switch_byte
ROM:B5DE
ROM:B5E0

The same kind of routine also exists for a word parameter instead of a byte.
Most of the time, the case values will follow some kind of logical order, so if you see a bunch of sequencing bytes of word, interlaced with what look like offsets, and preceded by an lcall, you might want to chack what's on the other end of that lcall.

OR, the preferred way once you have made your initial pass at identifying code, look for a function that starts by popping DPH and DPL, and seek all the lcalls that cross reference to it to identify the switch tables.

Oh, and for those who might wonder, of course, as soon as you pop the PC address that's been enqueued on the stack, the lcall never returns, and becomes exactly like a jump.

Coming next: How the hell are these bloody strings and other data sitting in standalone data sections referenced, where there does not appear to be any obvious address referencing to them anywhere in the disassembly...

Now I remember why I hate the 8032...

2009-08-24T16:49:00.015+01:00

...and pretty much EVERY CPU designed by intel, ever.

It's not like they could have turned a 16 bit CPU into a 24 or 32 bit addressing one, because that's way harder to do than extending the 32 bit x86 to 64 bit for instance (or the original 16 bit x86 to 32 bit)...

"Look Ma! It's a flock of randomly chosen flying search engine terms to direct a certain kind of people here!": Mediatek, MT8226, 8226, LCD, Samsung, 8032, 8051, MT13x9, bank, page, 0xFFFF, reverse engineering, firmware.

The usual trick, apparently originally devised by Philips, is to use one of the 8032 ports (usually P1) as an extention of the 16 bit address bus, thus acting as a ROM bank selector => if you use 8 bits of that port, you now have 24 bit addressing.

Of course, this doesn't mean that your PC will suddenly become 24 bit aware, and since it still remains 16 bit, you need to have overlap code, for each of your 64 KB banks ([0000-FFFF]) so that, as far as the CPU is concerned, the same bank is still being used.

This usually results in LOADS AND LOADS of ROM space wasted with the following kind of crap:

ROM:8301 ; --------------------------------------------------------------------------
ROM:8301                 mov     DPTR, #0x7E2D
ROM:8304                 ljmp    bank_0E
ROM:8307 ; ---------------------------------------------------------------------------
ROM:8307                 mov     DPTR, #0xE4D1
ROM:830A                 ljmp    bank_0B
ROM:830D ; ---------------------------------------------------------------------------
ROM:830D                 mov     DPTR, #0xF51E
ROM:8310                 ljmp    bank_0F
ROM:8313 ; ---------------------------------------------------------------------------
ROM:8313                 mov     DPTR, #0xFA0D
ROM:8316                 ljmp    bank_0F
ROM:8319 ; ---------------------------------------------------------------------------
ROM:8319                 mov     DPTR, #0xF924
ROM:831C                 ljmp    bank_0F
ROM:831F ; ---------------------------------------------------------------------------
ROM:831F                 mov     DPTR, #0xFDA7
ROM:8322                 ljmp    bank_0D

with the bank switching function of the like:

ROM:8288 ; ---------------------------------------------------------------------------
ROM:8288 bank_0E:                                ; CODE XREF: ROM:84F6j
ROM:8288                                         ; ROM:852Cj ...
ROM:8288                 mov     A, P1           ; Port 1
ROM:828A                 anl     A, #0xF         ; 8 bits for high addressing => 16 banks
ROM:828C                 cjne    A, #0xE, ROM_8291 ; bank 15 (0x0E)
ROM:828F                 clr     A
ROM:8290                 jmp     @A+DPTR
ROM:8291 ; ---------------------------------------------------------------------------
ROM:8291 ROM_8291:                               ; CODE XREF: ROM:828Cj
ROM:8291                 swap    A
ROM:8292                 rr      A
ROM:8293                 push    ACC             ; Accumulator
ROM:8295                 mov     A, #0xBF ; '+'
ROM:8297                 push    ACC             ; Accumulator
ROM:8299                 push    DPL             ; Data Pointer, Low Byte
ROM:829B                 push    DPH             ; Data Pointer, High Byte
ROM:829D                 ljmp    ROM_BF70
ROM:82A0 ; ---------------------------------------------------------------------------
(...)
ROM:BF70 ; ---------------------------------------------------------------------------
ROM:BF70 ROM_BF70:                               ; CODE XREF: ROM:829Dj
ROM:BF70                 anl     P1, #0xF0       ; Go trough bank 0
ROM:BF73                 orl     P1, #0xE        ; Set the high address lines to bank 0x0E on Port 1
ROM:BF76                 ret
ROM:BF76 ; ---------------------------------------------------------------------------

As indicated above, each bank has to DUPLICATE this whole bank switching business at the exact same address. And there will be a short period where code is actually ran in Bank 0 (right after the "anl P1, #0xF0") before switching to the actual destination bank (here Bank 0Eh)

Only thing I can say is, seeing how crappy a CPU it is (don't get me started about how poorly optimized all the 8032 code I see seems to be), licensing for the 8032 must come really REALLY cheap!

But then again, having SoC makers like MediaTek design their systems on the cheap might have its advantages...

Why does windows find the need to modify the partition table at boot...

2009-07-23T17:14:00.006+01:00

...or how I almost lost 1.5 TB of data today.

If you use software/fake RAID, this might happen to you too.

See, this morning I thought I would upgrade my HTPC's BIOS just for the heck of it. This is the PC that has a 2TB RAID5 array based on nVIDIA's MediaShield.
Now, despite what I had selected during the BIOS update, the BIOS settings and DMI data were reset after reboot, which means that the HDDs were back to individual IDE emulated drives, rather than members of the RAID array.

Normally, this wouldn't be a big deal, except that, before I cancelled the Windows boot, it was apparently able to look at the disks (using the MediaShield driver), find out that the capacity of the disk it was booting from (now a single 1TB IDE/AHCI HDD) was less than the capacity reported in the partition table, and re-write the partition table of HDD1 to reduce the dimensions of the last partition.

Of course, re-writting a partition table without anybody asking you to is the shortest way to screw up a disk or RAID array, and screw up it did: As soon as I restored the RAID settings in the BIOS and booted Windows, my 1.5 TB data partition was now identified as unformatted and gone! Talk about massive data loss...

No respectable O/S should ever modify a partition table without asking the user first. It's just common sense: The O/S is never, and I have to stress out that part, NOT EVER, smarter than its user (no matter what the O/S developers might think, or how smart they think they are themselves). You do not modify a partition table without asking, EVER, it's really simple as that!

Now, after much cursing, and some accidental good luck, I found that if the first drive was disconnected from the RAID5 array (which happened accidentally as I was trying to invert HDD#2 and HDD#3, since it originally looked like the BIOS upgrade has modified the SATA IDs), the rest of the array booted fine, albeit in degraded mode, and saw the old 1.5 TB data partition alright. Definitely makes sense with the fact that Windows would of course only have modified the partition table of the boot drive while the HDDs were in IDE mode.
But of course, as soon as you remove one drive from your RAID5 array, and boot in degraded mode, the array will flag that drive as failed on next reboot

From there on, the solution is to re-add the drive to the array to resync. Takes a while, but if you trust your other disks not to fail duing the super-lengthy re-sync, probably the safest solution.
Otherwise, it's probably a good idea to have a copy of the Master Boot Record (i.e. the first 512 bytes) of every single drive from your array, and restore it using a decent O/S like Linux. Plus, as experience will show you time and again, it's also always good practice to keep a copy of the MBRs of all your disks that contain important data, so that you can try to address any kind of partition formatting catastrophe.

Displaying International Domain Names in the browser's address bar

2009-07-20T10:03:00.004+01:00

In IE (>7): Tools -> Internet Options -> Advanced -> International -> Always show encoded addresses -> Uncheck
A typical "All or nothing" Microsoft solution to a problem that demands a smart approach. Well done Redmond!
In Firefox (from Marc Blanchet's post): about:config and filter using keyword "IDN". There you have two possibilities:
1. Create a "network.IDN.whitelist.com" boolean key and set it to true => will display all the .com IDNs using their Unicode characters, a.k.a. "the Microsoft way"
2. Create a new network.IDN.whitelist boolean entry for your full domain (eg: "network.IDN.whitelist.xn--abc.com") and set it to true.

Of course, none of this is really satisfying, as domains that do not present any risk of homograph attack (eg: sequence of glyphs that are not used in any language, or one glyph characters that cannot be mistaken for any other), are still not displayed as Unicode by default in those browsers, which really sucks.

The way to address this issue would be for unicode.org to maintain a list of lookalike characters (i.e. characters considered as dangerous to use in an IDN because they can easily be mistaken for another character). The browser could then use this list (along with a list of all the valid Unicode characters at the time the list was created, as you want to take provision against new spoofable Unicode characters) to sort this issue.

Of course, Unicode is likely to have cold feet about producing such a list, as they could end up being sued if they miss a lookalike and end up being sued as a result, but really, as the authority, it should be part of their role...

Unicode charts as a single image

2009-07-16T19:26:00.004+01:00

Well, one guy was crazy enough to create a single image containing all the Unicode charts from unicode.org and put it online, and he was even nice enough to provide both an interface to dynamically visualize the rendered image, as well as allow people to download the whole thing (which, even if slightly outdated, I find super useful).

Props to Ian then!

Update 2009.08.24 - You might also be interested in this

Windows Command Prompt in UTF-8

2009-07-15T11:51:00.003+01:00

Change the default font to Lucida
issue the command chcp 65001

Convert a grayscale or RGB layer to an alpha channel in Photoshop CS 4

2009-06-15T11:30:00.023+01:00

Goddammit! I don't know if 3D graphics designer use Photoshop or not, and if they do, how they manage to put with it all day, but talk about a product not being intuitive... On the other hand, it's true that I know a thing or two about how Adobe doesn't care that much about any of the software they design to be either customer friendly or intuitive...

So, today, in our new "What we thought we'd do in about 30 seconds in Photoshop CS4 but ended up doing in hours" saga, we'll see how to convert a shadow effect layer into an alpha channel.

The purpose here is to display a nice transparent text overlay texture, with a blending shadow against a non uniform background in a 3D animation (eg. DirectX, OpenGL, ...). And because I'm in a generous mood today, I'm even gonna drop some illustrations here in there.

Let's say that you are creating an animated "motivational poster", like the one on the left, and your goal is to do some nice animation of the motto like a zoom out for instance, while making sure the motto has a shadow for added impact. You will of course break down your animation between a static background and a text layer, loaded as a texture, and animate the latter.
Now, if you merely save the text layer with a shadow effect in Photoshop and use that as your texture, it will not blend nicely at all, as the shadow will be a solid colour, and eat part of the the nice aperture green logo. Therefore you want to add an alpha component to your texture, which you'll use for shadow blending.

At this stage, I'll suppose that you have your nice text layer with its shadow effect. The good part about all this ordeal is that you won't have to create a separate layer out of the effect, or play with adding/substracting selections, because a black text display with its shadow is pretty much exactly what you need to use for the alpha channel. The only thing is, if you use coloured or texturized text, you'll want to convert it to solid black before the next operations, through a duplicate layer or something.

Now, if you're a Photoshop novice and you've done your research, you should be aware that the alpha components of an image is dealt with as a channel in Photoshop, and that there is a tab for that (again non-intuitive as hell: where my "right click -> new channel" option, Adobe? And how am I supposed to know what the small icons at the bottom represent when there is no bleeping contextual help in your product?!?). So you'd think that, with all the grpahics people dealing with channels, depths masks and alpha all day long, there'd be a bloody "Convert Layer to Channel" function readily available by default in the interface. Well, if it was straightforward to work with alpha channels in Photoshop, that would make the life of graphics designers too easy, and obviously this is not something that you would ever want. So, after cursing for about 10 minutes about those non-intuitive menus, we have to fallback to looking on the internet for some helpful souls (and I got to give it to CreativeCow there, for providing all the actual solution steps):

Search into your Photoshop installation directory for .atn files (AcTioNs) and locate the one labelled "Video Actions.atn". Mine was in "D:\Program Files\Adobe\Adobe Photoshop CS4 (64 Bit)\Presets\Actions"
In Photoshop, go to Window -> Actions. You will see a set of "Default Actions", but of course, not all the Actions that come with Photoshop have been loaded by default in one of the more obscure menu of the product, where noone would mind if they had been.
Then click on what has to be the most useful button for advanced users, and therefore the one that's been made as easy to overlook as possible, namely that bleeping contextual dropdown menu.

And I really have to wonder what is it with this late redesign of UIs by Adobe and Microsoft (eg. Office 2007), whose only purpose seems to make the power users' life a living hell by hiding the most powerful feature in a place where the sun don't shine no more...
In that menu, you will find a "Load Action" option, which you should use to navigate to the folder where you spotted the "Video Actions.atn" file, and open that file.
Alternatively, you can skip step 1 and directly select "Video Actions" in that menu to make the Video Actions appear. Why "Video Actions" is in that menu but not in the main action window by default is really beyond me here...
OK, now we're talking: In the "Video Actions" section, the first 2 actions are: "Alpha Channel from Visible Layers" and "Alpha Channel from Visible Layers (inverted)". The Ark of the Covenant at last! And all we had to do was get our face and soul melted for it - a bargain! Merrily we go then by setting only our "black text with shadow" layer visible (make sure you don't select the background) and then clicking the little "Play" button in the Actions menu with "Alpha Channel from Visible Layers" selected. You should end up with a new channel that looks like the one on the left. In this image, white is for pixels with no alpha (completely opaque) whereas black is for pixels with 100% alpha (completely transparent).
Now, before we save our image with its newfound alpha channel, we want to lay it over black solid background for the shadow, so just create one and save only the text layer (with or without the shadow effect - it shouldn't matter) over the black background as your image with alpha. You might want to enable the alpha channel (set the eye) after creating the black background to have a better idea of what exactly you're saving.
Now because there does not appear to be clear conventions as to what a zero alpha value is meant to represent (this can vary from application to application), you might find that the alpha mask you get in your application does the opposite effect of what you wanted. If that,s the case, rather than redo the steps using the "Inverse" function for layer to alpha we've seen in the menu, you can simply double click on your Alpha channel in Photoshop and toggle between "Color Indicates Masked Area" / "Color Indicates Selected Area", as this will invert your Alpha channel

And that, sir, is how you do it. Of course, if you look at the actions, you can probably figure out how to achieve the same from the various menus (haven't really figured that one just yet, especially where a on/off selection is converted to a grayscale range). I also haven't figured out how you can insert one of those actions into the standard Photoshop menus (wouldn't it sit nicely in the layer menu?).

Oh and finally, if like me you've cursed Photoshop's stupid and uninformative handling of undo (Why is it that a 10 year old program like Paint Shop Pro 5 does a much better job at undo than Photoshop?), note that you can access the undo history from the Window menu as well, where you will actually get a description of the actions you have taken.

More vi search and replace

2009-04-20T11:31:00.002+01:00

To replace ^M characters that did not quite make it into carriage return when displayed in vi:

:%s/<Ctrl-V><Ctrl-M>/<Ctrl-V><Enter>/g

Note that <Ctrl-V> is a magic sequence that allows vi to uninterpret the next special key, inserting its control code instead.

redhat/centos services and mdadm quicknotes

2009-03-12T12:36:00.002Z

Always seem to forget stuff there. The command I'm looking for to manage services/change runlevel on redhat/centos is chkconfig (chkconfig --list).

Also, to re-add a partition to an existing (failed) RAID array:

mdadm /dev/md0 --manage --add /dev/sdb1

Mysql logging

2009-03-09T23:27:00.005Z

One day or another, you'll want to check what exactly your MySQL server is doing (to check for SQL injections for instance). Therefore:

To log all SQL queries, add the following to your my.cnf (in the [mysqld] section)
```
log=/var/log/mysqld.log
```
To log queries that actually modify the database:
```
log-bin
```
If you don't modify the default log-bin parameters, this would create a <servername>-bin.###### & index in your /var/lib/mysql, which you can then analyze with mysqlbinlog

Getting some bloody files from Gigabyte

2009-03-09T15:52:00.004Z

There's not ONE motherboard manufacturer to save the others. NOT ONE!
Listen, we all know that you're hardware developers, and that providing files is an afterthough, but that is no reason to:
- force customers to use God-awful, slow, and completely useless interfaces on your website when all they want is access BIOSes or drivers
- host your bloody files on a server that must still be powered by DX2 66MHz CPU with 8 MB RAM. Yeah, it really looks that slow from our end!!!
- suddenly decide that you're gonna change all the files location, because it was a bit too convenient for customer to google and access the data directly, so what better way to say "f*ck you!" than changing the interface and means of accessing the files every 6 months or so.

Asus, Gigabyte, MSI, Supermicro, whatever: YOUR SUPPORT & DOWNLOADS WEBISTES ALL SUCK, BIG TIME!!!

It's not like we're asking for much. We just want to access the BIOS, drivers and utilities without having to jump in slow motion (or should I say bullet-time, considering how time crawling to a standstill page refreshes appear to be) through hoops. And I won't hold you much of a grudge for designing websites that suck, because that's not your main area of expertise, but having them run on underperforming hardware is hardly the best way to put your customers in confidence.

All this rant to tell you that, if you actually want to download a file from GigaByte, don't bother going to gigabyte.com.tw. Just use ftp://download.gigabyte.ru/

isw: Could not find disk /dev/sd# in the metadata error with dmraid

2009-02-26T13:36:00.004Z

Reinstalled my old rig yesterday, to switch from a RAID0 to a RAID1 conf, now that the bulk of my data is stored in the RAID5 array from my HTPC. As usual, using the Motherboard's fakeraid driver. This time it's the intel's ICH8R Matrix Storage RAID driver (and boy is it slow on rebuilding RAID1 arrays, which I had to do since I only got a brand new Vista64 stable for four hours before a complete freezout that required a hard reset!).

As usual, went through a Linux reinstall, only to find out that dmraid 1.0.0.rc15 would return the following:

root@dusk:/usr/src/dmraid/1.0.0.rc15# dmraid -ay
ERROR: isw: Could not find disk /dev/sdc in the metadata
ERROR: isw: Could not find disk /dev/sdb in the metadata
no raid disks

Well, once again, it was Debian patches to the rescue. Just make sure you apply the latest dmraid_######.diff patches to your dmraid install (just like we did for grub, see below) and voila!:

root@dusk:/usr/src/dmraid/1.0.0.rc15# dmraid -ay
RAID set "isw_dadaadaecb_Ark" was activated
RAID set "isw_dadaadaecb_Ark1" was activated
RAID set "isw_dadaadaecb_Ark2" was activated

Booting with a failed primary HDD on a Linux RAID1 array

2009-02-24T12:58:00.003Z

Thought it would be straightforward, and that the system would automagically take care of it by itself, think again:

GRUB is dumb, until we tell it exactly what to do, so, assuming your /boot partition is located on sda1/sdb1, you need to make sure that you manually setup both disks, using:

[root@whatever ~]# grub

    GNU GRUB  version 0.95  (640K lower / 3072K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename.]

grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  15 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd0) (hd0)1+15 p (hd0,0)/boot/grub/stage2
/boot/grub/grub.conf"... succeeded
Done.

grub> root (hd1,0)
 Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd1)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd1)"...  15 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/boot/grub/stage2
/boot/grub/grub.conf"... succeeded
Done.

grub> quit

If you have additional disks that you fsck (last columns of your fstab), be mindful that libsata will reorganize their names around, so if you set your /dev/sdc1 to be checked at bootime, you might end up getting stuck with fsck not finding the disk, and not being able to boot at all!

Booting fakeraid RAID5 Linux, the less half assed way

2009-02-10T12:39:00.012Z

This will be the final episode of our ongoing installation of Linux on a nVidia MediaShield fakeraid RAID5 array.

By now, you should have a kernel and an initrd.gz image that allow you to boot onto your RAID5 partition, the only last remaining problem being that you need an external non RAID5 disk to do so. That disk could probably be put to better use, so let's remove it from the equation. And no, we're not gonna use an USB stick, CD-ROM or even a floppy, because what we need is a solution always at our fingertips, that we can rely on 24/7, without having to look for a flimsy little device/disc, that we're sure to lose or scratch anyway. Instead, we're going to use something MUCH better.

Re-introducing, the best invention since sliced bread: the fully hacked WRT54G, with supporting cast: PXE!

If by now you don't have a hacked WRT54G in your house, you're an idiot, and if you didn't get a motherboard + ethernet controller that supports PXE, you also need to rethink your life.

You'll find plenty of (seriously confusing) guides on how to use PXE, so I'll just cut down to the chase. What we need here is a PXE executable that can boot our vmlinuz + initrd.gz, as well as relinquish the boot operation to the default disk device, so that we can chose between Vista and Linux.
If you do the groundwork, you'll probably find out that GRUB can be compiled to be run from PXE. The only issue we have here is that the nForce forcedeth network driver is not embedded with GRUB 0.97, and while you can pick one up from gPXE (which is where all the network drivers from GRUB come from anyway), if you want to do things clean, you'll have to recreate the configure/Makefile settings to add the forcedeth, which is a major pain in the ass (or you can simply hijack one of the existing drivers files, and replace it with the forcedeth.c code, but this lazyman's option of choice is not really gratifying).

Instead, we're going to use PXELINUX, a spinoff of the Syslinux project.
Since our WRT router is using DNSMasq, we'll practically follow the instructions from the link above, so if you need more than what I am to present below, feel free to refer to the official documentation.

Create a tftpboot directory on your multipurpose router. For convenience, I am creating mine in /share/tftpboot as the /share directory is already shared using Samba, and this of course gives us great convenience to setup our PXE server files
Copy your vmlinuz and initrd.gz images in this directory
Download and extract the latest syslinux.zip from kernel.org.
Copy the pxelinux.0 from the syslinux.zip core/ directory into your tftpboot dir
Copy menu.c32 from /com32/menu into your tftpboot dir
It's also a nice feature to be able to boot DOS floppy images using PXE as well (eg: Win98 bootdisk, Memtest86+, etc.). To do that, copy memdisk from the memdisk/ directory to your tftpboot dir. Note that memdisk can also handle gzip compressed images, which can help reduce transfer time
Optional: If you want to confirm that memdisk works, or if you simply want to have the Memtest86+ option, you can also download the compressed Memtest86+ v2.10 image that I am providing here. Note that this is the same memtestp.bin that you can download from http://www.memtest.org/ except padded to reach the standard 1.44 MB floppy size. Despite multiple attempts, I haven't yet found how to get memdisk to auto-pad floppy images...
Oh, and the reason I use Memtest86+ rather than Memtest86 is that the latter didn't seem to handle my 4 GB RAM properly
Create a pxelinux.cfg/ directory in your tftpboot dir

Create a graphics.conf text file there, with the following content:

menu color tabmsg 37;40      #80ffffff #00000000
menu color hotsel 30;47      #40000000 #20ffffff
menu color sel 30;47         #40000000 #20ffffff
menu color scrollbar 30;47   #40000000 #20ffffff
menu master passwd yourpassword
menu width 80
menu margin 22
menu passwordmargin 26
menu rows 6
menu tabmsgrow 15
menu cmdlinerow 15
menu endrow 24
menu passwordrow 12
menu timeoutrow 13
menu vshift 6
menu passprompt enter password:
noescape 1
allowoptions 1

Create a default text file in the pxelinux.cfg directory, and adapt the following content to your needs (NB: By default, this menu will apply to all PXE devices. If you want to target specific MACs, just change the "default" name to the MAC address of your interface (eg: "01-88-99-aa-bb-cc-dd"):

default menu.c32
prompt 0

menu title PXE Boot Menu
menu include pxelinux.cfg/graphics.conf
menu autoboot Starting Local System in # seconds

label bootlocal
menu label ^Local System
menu default
localboot 0
timeout 35
    
label linux
menu label Slackware 12.2 (RAID5)
kernel vmlinuz
append initrd=initrd.gz vga=791 vt.default_utf8=1

label dos
menu label DOS (Win98)
kernel memdisk
append floppy initrd=win98.gz

label memtest
menu label Memtest86+
kernel memdisk
append floppy initrd=memtestp.gz

Now, you need to tell your DHCP server that it should handle PXE, and which files it should provide. If you are using DNSMasq, this is as trivial as adding the following lines:
```
# PXE Booting
enable-tftp
tftp-root=/share/tftpboot
# This is the file that will be ftpd's accross
dhcp-boot=pxelinux.0
```
and then restarting the daemon (/etc/init.d/dnsmaq restart)
Finally, you can enable PXE in your PC BIOS, and watch in awe as you can now conveniently boot whatever you want from the network. If it doesn't work, I'd suggest you run a test with memdisk and a bootdisk (eg: memtest), as it shouldn't require much of any configuration to run properly.

RE: Installing XBox controller on Vista 64

2009-02-10T12:29:00.005Z

Well, I already gave you some tips, but each time I do it, it's a complete pain finding the right driver and getting it to work. I'll tell you; there are WAY TO MANY versions of the xbcd driver floating on the internet.

The one you want to use then is the one listed from this thread (0.2.6 at the time of this article - I also put a copy of the installer here if you need). Don't bother installing anything else.
Then the nice thing is that this latest version is already self signed, and the installer will also prompt you to install the driver's root CA, so if you're in test mode, you're golden.

2 things to know though:
- To access the setup utility, you need to have disabled UAC (use msconfig for that). You can re-enable it afterwards
- If you already installed an xbcd driver, or if it simply doesn't appear to work, just select "update driver" on the device, and point it to your Windows\Inf directory. The update should get you sorted.

Booting fakeraid RAID5 Linux, the half assed way

2009-02-06T15:58:00.015Z

OK, by now you have a whole Linux system sitting idly on one of your RAID5 device mapper partitions. First thing you want to do then is edit the etc/fstab there to have the root filesysten point to the right device (in our case /dev/mapper/nvidia_aeejfdbep2). Not that it's actually necessary considered how we are going to boot, but it can't hurt.

If you've done your groundwork, you found out by now that using GRUB or LILO as is won't be of much help, as none of them is able to handle a RAID5 device mapper array. We don't have a choice her but have to run dmraid -ay before we try to access the disks, and of course, that means we need to build an initrd image.
As a side note, RAID1 or RAID0 shouldn't be an issue with GRUB, so you can probably follow this tutorial and complete the bootloader installation on your main RAID5 array, and have it work.

Now, our final solution here will still require an external partition with a /boot directory that the bootloader can refer to at boottime, but for the sake of this exercise, and also for our final solution, which will no longer require an external non RAID5 partition (more about that in the next post), we're gonna setup our RAID5 Linux system as close to standalone as possible, which means that what we ultimately want is the ability to run the bootloader from our RAID5 system (so that the day we have a bootloader that properly handles RAID5, we can just install it on the RAID5 MBR rather than the external disk, and ditch the latter in a heartbeat).
Unfortunatley, this means that we'll need to use GRUB rather than LILO, so we'll break things in 2 parts: First we'll configure initrd and setup LILO from the non RAID Linux to boot our RAID, and once we're there we'll setup GRUB to boot our RAID from the RAIDed Linux (but still using the non RAIDed boot). If you're confused, just hang on.

Part 1: Setting up initrd to boot the RAID5 Linux

Well, time and time again, I find more compelling reasons to use Slackware, the last one being /boot/README.initrd, which is installed by default, and in which Patrick Volkerding tells you anything you want to know about how to create initrd. Not that there is much you need to know in the end as:

root@stella# cd /boot
root@stella# mkinitrd -c

is all you need for now. The mkninitrd will have created a brand new initrd-tree and initrd.gz for you - isn't that nice?

Now, obviously, we need to add our dmraid executable to the initrd-tree and recreate initrd.gz. But if you're thinking "fine, we'll just pick up the exec", think again!
The dmraid executable we built was the dynamically linked version, so if you strace the files used, you'll see that we're gonna have to copy a whole bunch of libraries as well:

root@stella# strace -e trace=file dmraid -ay 2>&1 | more
execve("/sbin/dmraid", ["dmraid", "-ay"], [/* 35 vars */]) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/lib/libdevmapper.so.1.02", O_RDONLY) = 3
open("/lib/libc.so.6", O_RDONLY)        = 3
open("/proc/mounts", O_RDONLY)          = 3
(...)
RAID set "nvidia_aeejfdbe" already active
RAID set "nvidia_aeejfdbep1" already active
RAID set "nvidia_aeejfdbep2" already active
RAID set "nvidia_aeejfdbep3" already active

NB: The removed output above has to do with /proc, /dev or /sys, which won't be an issue.

The smart way (or lazy way, which is even better), is to compile dmraid as a static binary, so that we don't have to care about those pesky libraries. Therefore:

root@stella# cd /usr/src/dmraid/1.0.0.rc15/
root@stella# make clean
root@stella# ./configure --enable-static_link
root@stella# make
root@stella# ls -alF tools/dmraid /sbin/dmraid
-rwxr-xr-x 1 root root 204658 2009-02-05 23:01 /sbin/dmraid*
-rwxr-xr-x 1 root root 816664 2009-02-06 16:47 tools/dmraid*
root@stella# cp tools/dmraid /boot/initrd-tree/sbin/

That's 600 KB more to our initrd right there, but at least we know that we have everything we need.
Now, all that's left is editing the init script in /boot/initrd-tree to call our command.
Depending on your distro, the name & content of the init script could be very different, so you might have to be creative. In the case of slackware, the init script is called "init" (which makes sense, because then you don't have to specify it as a kernel parameter), and in the

if [ "$RESCUE" = "" ]; then

section, which already contains some disk detection routines, we're gonna add:

  # Initialize DMRAID:
if [ -x /sbin/dmraid ]; then
/sbin/dmraid -ay
fi

Now, we shall rebuild our initrd:

mkinitrd -r /dev/mapper/nvidia_aeejfdbep2

The stage is now setup to see if we can boot our RAID5 system using LILO, by adding the following section in /etc/lilo.conf:

image = /boot/vmlinuz
initrd = /boot/initrd.gz
root = /dev/mapper/nvidia_aeejfdbep2
label = RAID5_Linux
read-only

Now, reinstall LILO:

root@stella# lilo
Warning: '/proc/partitions' does not match '/dev' directory structure.
Name change: '/dev/dm-0' -> '/dev/disk/by-name/nvidia_aeejfdbe'
Warning: Name change: '/dev/dm-1' -> '/dev/disk/by-label/Vista64'
Warning: Name change: '/dev/dm-2' -> '/dev/disk/by-name/nvidia_aeejfdbep2'
Warning: Name change: '/dev/dm-3' -> '/dev/disk/by-label/Media'
Added RAID5_Linux *
Added Slack_New
Added Slack_Old
Added Rescue_1
4 warnings were issued.

Don't worry too much about the warnings. Just reboot and, yay, it works!... Err, well... kinda, because while you should have seen the dmraid disks being mapped, and you do end up with everything running as it should from the RAID5 partition, you might end up with a boot where the rc.d/init scripts from Slackware are not being displayed on the console at bootime as they should.
If you look in /var/log/messages, you will see that all the scripts do indeed run, but you're being left with a very silent screen right before you end up with the prompt.

The problem is actually due to udev screwing up the /dev repository after the root filesystem is mounted. The solution to that? Keep the /dev used by the kernel even after root is mounted by compiling your kernel with:
Device Drivers ---> Generic Driver Options ---> "Create a kernel maintained /dev tmpfs (EXPERIMENTAL)" and "Automount devtmpfs at /dev" bot selected.

Part 2: Setting up GRUB on the RAID5 Linux partition

There are 2 (well 3) reasons why we want to replace LILO with GRUB here:
1. GRUB is more likely to be patched for full RAID5 support compared to LILO, so when that happens, we want to be ready
2. If you're using RAID0/RAID1 instead of RAID5, GRUB should actually be able to install the bootloader on your RAID array
3. Having GRUB handle RAID takes some trickery which you probably want to read about.

Once more, we'll be using grub 0.97, however, if you use the vanilla version, no matter what you do or where your /boot partition might be located (even on a standard non-RAID disk), you might end up with the infamous:

grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... no

Error 2: Bad file or directory type

AFAIK, this could be due to GRUB 0.97 being unable to access ext3 256 bytes inodes, OR it could happen if you have more than 2 GB RAM, or it could have to do with the 2.6 kernel new geometry. Well, all I know is that one of the grub 0.97 patches from Debian fixes the problem. Thus:

4) Reinstall grub on RAID:
root@stella# wget ftp://alpha.gnu.org/gnu/grub/grub-0.97.tar.gz
root@stella# tar -xvzf grub-0.97.tar.gz
root@stella# cd grub-0.97
root@stella# wget http://ftp.de.debian.org/debian/pool/main/g/grub/grub_0.97-47lenny2.diff.gz
root@stella# gunzip grub_0.97-47lenny2.diff.gz
root@stella# patch -p1 < grub_0.97-47lenny2.diff
root@stella# cat debian/patches/00list

OK, that last line gives us the order in which we should do the installation of the patches, so from there on you just need to run a bunch of:

patch -p1 < whatever.patch

In the order provided from the 00list file (i.e. starting with cvs-sync.patch and ending with use_grub-probe_in_grub-install.diff). Once you're there, just compile and install grub so that we move to the final phase. Now, we'll still tell GRUB to use our /boot directory on /dev/sdc1, because we don't really have a choice here (if you don't believe me, you can try installing on RAID5 and see your 'setup (hd#)' command fail miserably), but we will also tell it how to "see" our RAID5 array, and to be able to do that, we will need to know our disk geometry, which we can get from fdisk. What we want are the C(ylinders) H(eads) and S(sectors) value:

root@stella# fdisk /dev/mapper/nvidia_aeejfdbe
Command (m for help): p

Disk /dev/mapper/nvidia_aeejfdbe: 2000.4 GB, 2000409722880 bytes
255 heads, 63 sectors/track, 243202 cylinders

These days, most disks have 255 heads and 63 tracks anyway (which are the greatest values you can set), so what you really need is the number of cylinders. This we will use to provide the geometry of our RAID5 "disk" to GRUB, in C H S order, because it is unable to figure it our by itself. And also, since we are doing a GRUB installation from scratch, we have to copy the stage1 & stage2 files to the /boot/grub directory (which is information that the clueless people using ready made packages are apparently unable to provide - damn you Ubuntu!):

mount /dev/sdc1 /mnt/hd
mkdir /mnt/hd/boot/grub
cp /usr/local/lib/grub/i386-pc/* /mnt/hd/boot/grub/
grub --device-map=/dev/null
grub> device (hd0) /dev/sdc

grub> device (hd1) /dev/mapper/nvidia_aeejfdbe

grub> geometry (hd1) 243202 255 63
drive 0x81: C/H/S = 243202/255/63, The number of sectors = -387927166, /dev/map
per/nvidia_aeejfdbe
Partition num: 0,  Filesystem type unknown, partition type 0x7
Partition num: 1,  Filesystem type is ext2fs, partition type 0x83
Partition num: 2,  Filesystem type unknown, partition type 0x7

grub> find /boot/grub/stage1
(hd0,0)

grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0x83

grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... yes
Checking if "/boot/grub/stage2" exists... yes
Checking if "/boot/grub/e2fs_stage1_5" exists... yes
Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  16 sectors are embedded.
succeeded
Running "install /boot/grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/boot/grub/stage2
/boot/grub/menu.lst"... succeeded
Done.

Good, now we're ready to create our boot menu:

vi /mnt/hd/boot/grub/menu.lst

default 0
timeout 3

title  Vista (64 bit)
rootnoverify (hd1,0)
chainloader +1

title  Slackware 12.2 (RAID5)
root   (hd1,1)
kernel (hd0,0)/boot/vmlinuz root=/dev/mapper/nvidia_aeejfdbep2
initrd (hd0,0)/boot/initrd.gz

This allows us to boot both Vista and Slackware on the RAID5 array using the /boot partition on the non RAID disk. Note that at this stage, it is probably a good idea to duplicate the /boot directory from the non RAID to the RAID partition, and recreate a small boot partition from scratch on the non RAID.

Of course, as mentionned before, because we still need a non RAID HDD, this is an half assed solution. In the next post, we'll see a better assed solution, where we do away with that extra HDD, and where we'll explore some new interesting stuff...

And here we go again - RAID5 fakeraid + Linux

2009-02-05T21:40:00.021Z

It's that time of the decade where I get a new machine, this time an nVidia MCP79 (a.k.a. GeForce 9400) based motherboard from Gigabyte. Oh, and yes, the MCP79 is the same all-inclusive nForce chipset nVidia uses in the new MacBook Pros, as well as their Atom "can of whoop-ass" for set top HDTV boxes.

As most people, I'm pleasantly surprised by the features of the chipset. My goal was to build an HTPC, and with one good durable mobo (with both HDMI and optical SPDIF), the adding of a 45 nm quad core and 4 GB dirt cheap DDR2 RAM, I should have a solution that will last for some time. Oh, and of course, since we're hopefully going to store a lot of media data there, we can't forget the 3x 1 TB Samsung SpinPoint HDDs in a RAID5 array, which is what brings us here today.

Yep, as you guessed from the title, it's ye olde getting Linux to work with fakeraid. And the first thing I gotta say is I'm sick and tired of all the posts you see on the net where people asking how to make Linux work with their fakeraid are met with a "why don't you use Linux's md instead".
Well, I'm sorry, but there are still people out there who want to dual boot with Vista and trust nVidia RAID5 driver (MediaShield) enough not to want to make a complete mess of their Windows installation. Hell, I'd rather spend time figuring out how to make my RAID5 work in Linux than have to spend time doing the same thing in Windows. Plus, I like the convenience of having my BIOS handle things like RAID.

Here we are then, with Vista installed and painfully updated at last (see below) and a blank partition on the nVidia RAID5 MediaShield array that's just dying to get Linux installed.

OK, it's not that I mind a challenge here, but I've got a blank HDD lying around, so this time, what we're going to do is install Linux on that standalone drive, conf it so that it sees the RAID5 partition, and then move over our system rather than have to create our own custom RAID5 installation CD.

As usual, I'm gonna use the latest Slackware (12.2) so ymmv.

1. Slackware install with ALL the options you really want in the end (we'll be using that partition as the source) and LILO booting the single HDD = piece of cake. Just make sure that you don't install LILO on anything else BUT the HDD that's not part of the array
2. First bad surprise - if you keep your 2.6 kernel as is, no matter what you try, dmraid will return:

ERROR: device-mapper target type "raid45" not in kernel

Don't waste your time changing the raid45 to raid456 identifiers in dmraid either - you'll still get the same error. The kernel actually needs to be patched for dmraid to be happy.

[UPDATED]One year later, and still a mess. As of 2.6.35, the dm-raid45 module still hadn't been integrated to the kernel, plus the evil people at Red Hat who were supposed to maintain that module have completely dropped the ball, the Gentoo people also have dropped the ball (after 2.6.31) and the last set of dm-raid45 patches I could only go to 2.6.33 (and don't work against 2.6.35). So I guess we're stuck with 2.6.33 then. This last set of dmraid patch can be found from Mandriva. Can't help but find it strange that, while the dm-raid45 module is commonly used in Ubuntu (except that dmraid itself is useless with large disks as it doesn't support GPT), nobody seems to care about trying to merge it with the official kernel tree. Oh well...

root@stella# cd /usr/src/linux-2.6.33/
root@stella# wget http://tmb.mine.nu/Mandriva/Cooker/dm-raid/dm-raid45_2.6.33-rc1-20091126.patch
root@stella# patch -p1 < dm-raid45_2.6.33-rc1-20091126.patch
root@stella# wget http://tmb.mine.nu/Mandriva/Cooker/dm-raid/dm-raid45-buildfix-for-2.6.33.patch
root@stella# patch -p1 < dm-raid45-buildfix-for-2.6.33.patch

OK, so you just go through your usual kernel recompile. Now, a couple of things I wanna point out:
- You do want to reduce the size of your kernel if you're planning to remove the standalone HD and still boot your RAID5 array Linux partition with a decent solution
- Of course you want to add Device mapper support (Device drivers -> Multiple devices driver support (RAID and LVM)) and the new experimental "RAID 4/5 target" that was added by the kernel patch
- Despite being a Gb adpater, the nForce Ethernet driver (forcedeth) is to be found in: Ethernet (10 or 100Mbit) -> EISA, VLB, PCI and on board controllers -> nForce Ethernet support
- if you're using ext4 for root, don't forget to include ext2 support as well, as your root partition cannot be mounted without it.

Once you have a kernel that boots, you can move on to the dmraid install. This time, no need for devmapper. Just download the current dmraid source from http://people.redhat.com/~heinzm/sw/dmraid/src/:

root@stella# cd /usr/src
root@stella# wget http://people.redhat.com/~heinzm/sw/dmraid/src/dmraid-current.tar.bz2
root@stella# tar -xjvf dmraid-current.tar.bz2
root@stella# cd dmraid/1.0.0.rc15
root@stella# ./configure; make; make install
root@stella# dmraid -ay
RAID set "nvidia_aeejfdbe" was activated
RAID set "nvidia_aeejfdbep1" was activated
RAID set "nvidia_aeejfdbep2" was activated
RAID set "nvidia_aeejfdbep3" was activated

Yay! it works!

If you only see see the RAW disk (nvidia_aeejfdbe) but not the partitions, that's probably because you're using a GPT disk. In that case, you need to also run kpartx as follows:

root@stella# kpartx -a -v /dev/mapper/nvidia_aeejfdbe
add map nvidia_aeejfdbe1 (252:1): 0 262144 linear /dev/mapper/nvidia_aeejfdbe 34
add map nvidia_aeejfdbe2 (252:2): 0 209715200 linear /dev/mapper/nvidia_aeejfdbe 264192
add map nvidia_aeejfdbe3 (252:3): 0 186478592 linear /dev/mapper/nvidia_aeejfdbe 209979392
add map nvidia_aeejfdbe4 (252:4): 0 11324626944 linear /dev/mapper/nvidia_aeejfdbe 396457984

kpartx itself can be obtained from the multipath-tools.

OK, so now we can get going. Be sure you format the RELEVANT partition. In my case, I'll be using nvidia_aeejfdbep2, the second partition on the RAID5 array for the GNU/Linux system.

root@stella# mkfs.ext4 /dev/mapper/nvidia_aeejfdbep2
root@stella# mount /dev/mapper/nvidia_aeejfdbep2 /mnt/hd
root@stella# mkdir /mnt/hd/proc
root@stella# mkdir /mnt/hd/sys
root@stella# cp -ax / /mnt/hd

This could take a while...
Note that the -x option ensures that we stay on a single file system

OK, in the next post we'll see the grub way to (kinda) boot this whole mess

These are not the ads you're looking for

2009-02-04T13:32:00.001Z

Here's a link to the hosts file I use on my Windows systems:
http://www.mvps.org/winhelp2002/hosts.htm

Windows update error 0x8024402c

2009-02-02T12:18:00.004Z

Well, like everybody else, this error has been driving me nuts. Plus it manifested itself in such a weird way I couldn't believe.

I already had that issue last week on a brand new Vista 64 install, and somehow, it appeared that resetting the LAN paramaters to default did the trick, and after a short hour struggle, I was able to apply the updates.

I was not so lucky with that old laptop of mine running XP, which was badly in need of critical updates, but that couldn't seem to download them. Well, actually, at first, it started to download the updates alright (got through about 8-10 of them) but then it failed! Tried every hint from the link above. None worked...

This is a job for... Is it a bird? Is it a plane? No, it's Wireshark!
OK, so Wireshark reveals that, despite everything Microsoft wants to make you believe, the issue is really with au.download.windowsupdate.com not being resolved. Of course, one has to wonder why an Irish based Windows has to pick up its updates from Australia, but hey, that's Microsoft for you.
Indeed, a few tests from the commandline show that au.download.microsoft.com is indeed not resolvable. OK, let's see what the DSL router (which can also do basic diagnostic checks) has to say about it... WTF? The DSL router sees no issue there:

Resolving au.download.windowsupdate.com ... 65.54.84.196
Reply from 65.54.84.196
Reply from 65.54.84.196
Reply from 65.54.84.196
Ping Host Successful

And this is where it gets really weird, as it appears that my Linux machines also cannot resolve the host... while the router can?!?! No way!

OK, by now I need to apply these updates badly, and therefore, yes, you got it, we're gonna edit the systems32/drivers/etc/hosts file of course and get done with it.
So here goes:

65.54.84.196 au.download.windowsupdate.com

and try downloading the updates again (with Wireshark still running in the background).
What the hell?!? No difference - We still get DNS requests for au.download.windowsupdate.com that don't seem to resolve. But how on earth is that possible with a triple checked hosts file?

The answer of course is that Microsoft deliberately screwed up their DNS resolution so that their update servers can NOT be overridden by the hosts file. Yeah, I can understand the logic of it, but how about relying on https and certificates instead, to check that the server is legit, rather than breaking down DNS. Talk about another half assed approach to security...

Well, that will teach me to try to work around a problem rather than solve it, because with no DNS override, I'm back to square one. And I'm not really willing to install my own DNS server just to work around this problem either.

Now, further use of nslookup with various Irish DNS seem to indicate that they ALL have the same issue??? Is someone out there to get me or what, because none of this makes any sense!

At this stage, instead of banging your head against the wall, you gotta try to apply logic. If all of the Irish DNSs were failing to resolve the Windows Update servers, you'd probably get people armed with pitchforks on the streets within minutes, so I must be doing something wrong.
A quick change of NS lookup tool actually confirms it (DON'T USE NSLOOKUP ANY MORE STUPID!). dig quickly shows that it's only the DNS that I was using (Eircom's 159.134.237.6 & 159.134.248.17 - never trust the incumbant!) can't resolve the bloody server, while my own ISP's DNS can.

A quick change of DNS servers on the DSL router, a full reboot (ipconfig /release + /renew would probably have done the trick, but with M$, you can never be too sure), and Windows Update is happy again.

Good thing I didn't have anything better to do this morning. And just to sum up the lessons for today:
- 0x8024402c happens if Windows cannot resolve the IP of the download servers
- Windows update CAN switch download servers on the fly while downloading (!)
- Use Wireshark to find out which download servers Windows Update is trying to use
- Windows name resolution is screwed so that it will always try to use DNS and not the hosts file for the download servers
- Don't trust what your DSL router tells you about Name Resolution - it might not be using the configured DNS servers!
- DON'T USE NSLOOKUP ANY MORE! Use dig instead.
- Screw Eircom!

This script will save your life!

2009-01-15T11:47:00.009Z

Or at least your data.

I have been unofficially sysadm'ing a bunch of HP DL380 running Linux over the past few years, and over the Christmas period, one of those, which had a complex mix of ReiserFS, SWAP and XFS partitions on a RAID5 Smart Array device (dev/cciss/c0d0), found nothing better than to start overwritting the MBR!

What this meant of course was, bye bye partition table, and, as you might guess, I didn't find the need to backup the partition data, thinking: "Heck it's RAID5 - what's the worse that can happen?"

ADVICE #1: If you're running a server in PROD, ALWAYS save a copy of the output of fdisk someplace safe!

Thus, there I was, with a non bootable server and a blank partition table, but with data I very much wanted to get back to. And the only tool I had at my disposal (through a remote console connection, because of course, the server had to be remote) was just a Slackware boot CD (because it detects cciss drives) running busybox.

ADVICE #2: If you have physical access to your server, and it's not using a nonstandard disk device, you can probably find a Linux rescue CD with gpart (but better use the latest Debian version, which is more up to date) that supports more partitions types, and will do a much better job. The script below is really if you are in a hurry or you have limited resources.

How difficult can it be to detect partitions with only a shell script then? Well not that difficult, as the script below will prove. But first let me be clear about what the script is meant to achieve:

This script is meant to detect the POTENTIAL beginning of a partition only. It will tell you the first cylinder, but it will not tell you the size, so unless it's the last one on disk, you'll have to figure out where the next partition begins as well
This script will detect primary partitions only. If you have extended partitions, you're on your own
The script only detected EXT2/EXT3, ReiserFS, Swap (version 2) and XFS. It does not detect FAT or NTFS (because I had no use for it). However, if you know the Magic string and location of other partitions types, you should be able to easily modify the script to add them
Not counting the comments, I kept the script short and basic, because you're most likely to have to type it in by hand, so shorter is better
Be mindful that there are likely to be false positives
And of course, while I succesfully tested this script and all the partitions types on various systems, I am not responsible for any damage occuring from using it.

OK, so below is the part you are interested in:

#!/bin/sh
#
#    gpart.sh v1.1 - Linux partitions detection script
#    Copyright (C) 2009  >NIL:
#    Based in part on gpart (C) 1999-2001 Michail Brzitwa et al.
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#

#
## Set your disk parameters below
#
device="/dev/sda"
cyl_start=1
# To find your max cyl_end, use fdisk or cfdisk
cyl_end=35697

#
## FS Magic length, string and offset
## Uncomment only one of the sections below
#

## XFS
#bs=4
#magic="XFSB"
#magic_offset=0x0

## ReiserFS
#bs=4
#magic="ReIs"
#magic_offset=0x10034

## Swap space (v2 only!)
#bs=4
#magic="ACE2"
#magic_offset=0x0FFC

## Ext2 / Ext3
bs=2
## OK, here I have to curse the EXT FileSystem devs for
## not chosing a *PROPER* ASCII Magic like everyone else,
## but using 0x53 0xEF instead as this contains i-umlaut
## Thank God for the -e option of echo, which translates
## a "\0###" sequence into the relevant octal character
magic=`echo -ne 'S\0357'`
magic_offset=0x0438

# on almost any recent disk, a cylinder is
# 255(tracks)*63(sectors/track)*512(bytes/sector) = 8225280
# If you're not sure, check what fdisk tells you
cyl_bytes=8225280
mbr_bytes=32256

#
## You shouldn't have to modify anything below this
#

# dd skips in multiples of bs, so we need to compute
# the cylinder size and magic_offset in bs blocks
cyl_blocks=$(($cyl_bytes / $bs))
mbr_blocks=$(($mbr_bytes / $bs))
magic_blocks=$(($magic_offset / $bs))

for i in $(seq $cyl_start $cyl_end); do
if [ $i == 1 ]; then
  # For the first cylinder, we need to skip the MBR as well
  skip=$(($magic_blocks + $mbr_blocks))
else
  skip=$(($(($i-1)) * $cyl_blocks + $magic_blocks))
fi
# Look for the magic block
header=`dd if=$device bs=$bs count=1 skip=$skip 2>/dev/null`
if [ "$header" = "$magic" ]; then
  echo "MATCH: cylinder $i ($header)"
fi
done

Incoming packet was garbled on decryption (putty)

2008-12-17T18:09:00.004Z

For some reason, I decided to install openssh on openWRT (Kamikaze), and I have been getting "Incoming packet was garbled on decryption" in putty. This seems to be a ongoing issue linked to the encryption cyphers used by openSSH on low resources systems.

The solution: In the Connection -> SSH menu of putty, make sure 3DES is at the top, and the problem will go away

mmc driver for openwrt, part deux

2008-08-12T18:43:00.007+01:00

Went over the MMC configuration for my openWRT router yesterday, as I wanted to relinquish the use of the SES button by the driver (GPIO4) to use it instead as a toggle button for the Wifi, and use GPIO5 for DO.
As I did just that, I also "fixed" the MMC driver to stop the Power LED from blinking every time the SD card was accessed.
The way I did that was to use the following config.h:

#define SD_CS_POWER 0x82

And the modify the spi.c to have:

static inline void mmc_spi_cs_high(void) {
port_state |= SD_CS_POWER;
(...)

Also, as I am now using GPIOs #2,3,5,7, I had to add a line

echo "0xac" > /proc/diag/gpiomask

in one of the init scripts.

For /etc/hotplug.d/button/01-radio-toggle, I am currently using the following, but I might change that to just turn the radio on/off:

if [ "$BUTTON" = "ses" ] ; then
  if [ "$ACTION" = "pressed" ] ; then
          WIFI_RADIOSTATUS=$(wlc radio)
#               touch /tmp/$WIFI_RADIOSTATUS
          case "$WIFI_RADIOSTATUS" in
          0|"")
                  uci set wireless.wl0.disabled=0
                  uci commit wireless
                  wifi
                  wlc radio 1
                  echo f > /proc/diag/led/wlan
          ;;
          1)
                  uci set wireless.wl0.disabled=1
                  uci commit wireless
                  wifi
                  wlc radio 0
                  echo 0 > /proc/diag/led/wlan
          ;;
          esac
  fi
fi

Despite the non-quite-thought-through use of the GPIOs in the default MMC driver, I'm beginning to think openWRT is the best invention since sliced bread!

Installing unsigned drivers on Vista 64

2008-08-10T20:13:00.015+01:00

OK, if you ended up here, it's probably because you've been trying to install an unsigned driver (eg. XBCD Xbox Gamepad, PSPLinkUSB), and found out about the requirement for all drivers to be signed in Vista 64.

Now, you shouldn't rush with the first article you found on the web that tells you how to disable signed drivers in Vista altogether. The MUCH smarter way is to run Vista 64 in test mode instead, and self sign your drivers. And to be clear, NO, this does NOT require you to recompile the drivers! You can just pick up the drivers you got from someone and sign them away. Of course, one could comment on yet another of Microsoft's stupid "we don't trust our users" decisions of having to enable the test mode to have users install their self signed drivers. A MUCH SMARTER way would have been to do that outside of the test mode as well. After all, if a user went as far as installing their own root certificate, it's probably that it should be trusted.

Anyway, the procedure is as follows (and it is described in much more details here):

1. Get Vista to boot in test mode always with the command:

bcdedit.exe /store C:\Boot\BCD /set testsigning yes

(And there again, I have to curse Microsoft for NOT indicating with bcdedit /? that you can use the /store option to specify your store, and having to spend HOURS trying to figure out why I was getting the following error which is apparently expected, if you boot multiple OSes and don't let Microsoft take over your boot record:

The boot configuration data store could not be opened.
The system cannot find the file specified.

)

After you enter that command, you MUST reboot Vista.

Note: Once Test Mode is enabled, you will get the Windows Version as well as "Test Mode" displayed over the background image. If you're bothered by this, what on earth are you doing with your computer? Staring at the background?

2. Download the necessary DDK SelfSign files, which I am CONVENIENTLY providing to you HERE, as Microsoft is also an ass there - People shouldn't have to download 2.7 GB to gain access to 700 KB worth of files!
Extract them to the directory where you have your driver

3. Let's say you want to install the PSPLinkusb driver. First you want to generate your own root certificate for that driver with:

makecert -$ individual -r -pe -ss "Self Signed Drivers" -n CN="Self Signed Drivers" selfsign.cer

4. Then you install the certificate you just created to the trusted root directory:

certmgr /add selfsign.cer /s /r localMachine root

(NB: if you have UAC on, you will need to run this command in a "run as administrator" command prompt)

5. Finally, you sign EACH .sys file using the certificate:

signtool sign /v /s "Self Signed Drivers" /n "Self Signed Drivers" libusb0.sys
signtool sign /v /s "Self Signed Drivers" /n "Self Signed Drivers" libusb0_x64.sys

Voila! Now you can install these drivers and get on with your life.

For completion, I am providing below the result of a successful certification for the libusb drivers:

E:\Program Files (x86)\OpenOCD\0.2.0\drivers\ft2232>makecert -$ individual -r -pe -ss "Self Signed Drivers" -n CN="Self Signed Drivers" selfsign.cer
Succeeded

E:\Program Files (x86)\OpenOCD\0.2.0\drivers\ft2232>certmgr /add selfsign.cer /s /r localMachine root
CertMgr Succeeded

E:\Program Files (x86)\OpenOCD\0.2.0\drivers\ft2232>signtool sign /v /s "Self Signed Drivers" /n "Self Signed Drivers" libusb0.sys
The following certificate was selected:
   Issued to: Self Signed Drivers
   Issued by: Self Signed Drivers
   Expires:   2040.01.01 00:59:59
   SHA1 hash: E0CEAD6474EFD1BF0F6D47501FF3F069C20FD7C7

Done Adding Additional Store

Attempting to sign: libusb0.sys
Successfully signed: libusb0.sys

Number of files successfully Signed: 1
Number of warnings: 0
Number of errors: 0

E:\Program Files (x86)\OpenOCD\0.2.0\drivers\ft2232>signtool sign /v /s "Self Signed Drivers" /n "Self Signed Drivers" libusb0_x64.sys
The following certificate was selected:
   Issued to: Self Signed Drivers
   Issued by: Self Signed Drivers
   Expires:   2040.01.01 00:59:59
   SHA1 hash: E0CEAD6474EFD1BF0F6D47501FF3F069C20FD7C7

Done Adding Additional Store

Attempting to sign: libusb0_x64.sys
Successfully signed: libusb0_x64.sys

Number of files successfully Signed: 1
Number of warnings: 0
Number of errors: 0

E:\Program Files (x86)\OpenOCD\0.2.0\drivers\ft2232>

No more "Access Denied" for your files on Vista

2008-08-04T15:41:00.008+01:00

Should really be in techno rant, because Vista's "security" is done completely backwards, but here we go. If you try to move/copy/delete/rename the file or directory C:\access_denied, this is what you need to do to recover access:

Open an admin command prompt and issue the following:

takeown /f C:\access_denied
icacls C:\access_denied /grant <username>:F

For what is worth, this is how you manage to delete the files from the infamous C:\Windows\System32\DriverStore\FileRepository, to stop Windows from being a BLEEPING BLEEP with the BLEEPING drivers. I'm gonna have the final say about WHICH driver I want to get installed dammit!

Script to convert Unix time to a human readable date

2008-07-17T10:26:00.003+01:00

unixtime:

#!/bin/sh
date -d @$1

adding a 2GB SD card on a WRT54G with openWRT

2008-07-10T23:07:00.006+01:00

It's been documented to death, so I'm not gonna bother telling you how exactly I did it. Just make sure you use the optimised mmc.o driver from http://forum.openwrt.org/viewtopic.php?id=9653.

I will however document the errors you can get when there's something wrong with your soldering, as I can attest that, if it doesn't work, it doesn't come from the card or the code, it comes from lousy soldering.

o [FATAL] mmc_card_init: invalid response from card: 03 found, waiting for 01

There is a short between your CLK line (usually GPIO3) and your DI line (usually GPIO2)
This can happen if your connection to the white LED is also touching the amber LED. TRIPLE check that your solder points are clean between the 2 front LEDs

o [WARN] mmc_init: impossible to get card indentification info for reason code: 01
o [FATAL] mmc_init: got an error when trying to get card configuration registers: 02

This one's a lot trickier. Just make sure you have the gpio tool at hand, and watch the amber LED closely. By default, it should be completely off. Then issue "gpio disable 3" and then "gpio enable 3". If you can now see a small residual light on the LED, that didn't exist before, then your LED is screwed and therefore your CLK is screwed.
Always make sure that your "enabled" voltages on GPIO are 0V and stay 0V.

I ended up removing the Amber LED (after a test using GPIO5 confirming that I had indeed an issue with GPIO3) and now everything works like a charm.

Oh, and if you're running in mixed mode (root fs still using the onboard flash), chroot is your friend to install packages on the card with ipkg.

Opening ports for remote access on ZyXEL DSL modem

2007-12-07T11:14:00.000Z

I have to do this once in a while, then completely forget how it's done, and one year later, I spend 1 hour trying to figure it out again. Well, at least on the Prestige 660RU-T1 it's fairly easy from the web interface: NAT -> SUA Only and add something like:

	Start Port No.	End Port No.	IP Address
1	All ports	All ports
2

Oh and just in case you wonder, the only mode of authentication allowed from the outside is private key, so good luck trying to break in!

Disable hibernation in Vista

2007-11-06T11:20:00.002Z

This can come handy if you have 4 GB RAM and plan to image your partition, as the last thing you want to archive is hiberfil.sys. From a command prompt, simply type:

powercfg -H off

and watch your hiberfil.sys turn to oblivion...

MY ASUS P5B Deluxe Linux (Slackware 11.0) settings

2007-05-13T02:03:00.003+01:00

Putting this here so that, if I ever have to reinstall Slackware in a hurry (as happened the other day), I can quickly revert to them, but first I gotta give you a handful of disclaimers
1. Because of ASUS & intel's new lousy 4 pin CPU fan control and the fact that my (whatever model) Zallman cooler only provides a 3 pin connector, my PSU is plugged to the CPU fan control, my CPU Fan is plugged to FAN2 or FAN3, and my side case fan is plugged to FAN1 or something. In short, my setup is nonstandard, so if you use my sensors.conf on a regular setup, your Fan labels will be screwed up (but all the rest should be OK)
2. Likewise, my fancontrol file is only meant to work for my rig! If you use it and fry yours, instead of running pwmconfig as you should, tough luck!
3. Kernel 2.6.21 has disabled one of the P5B Deluxe network adapters (the 88E8056), because the driver is reported as broken.
To re-enable it, look for "broken" in drivers/net/sky2.c

Here goes nothing:
o /etc/sensors.conf for MY Asus P5B-Deluxe

# Winbond W83627DHG configuration
# This is for an Asus P5B Deluxe Edition
chip "w83627dhg-*"
# Temperatures
label temp1 "MB Temp"
set temp1_over 50.0
set temp1_hyst 45.0
label temp2 "CPU Temp"
set temp2_over 60.0
set temp2_hyst 50.0
compute temp2 @+10,@-10
ignore temp3
# Fans
label fan1 "Side Fan"
set fan1_min 1000
label fan2 "PSU Fan"
set fan2_min 1000
label fan5 "CPU Fan"
set fan5_min 1000
ignore fan3
ignore fan4
# Voltages
label in0 "Vcore"
set in0_min 1.32
set in0_max 1.60
label in1 "+12V"
compute in1 @*(66/10), @/(66/10)
set in1_min 12.0 * 0.95
set in1_max 12.0 * 1.05
label in2 "+3.3V"
set in2_min 3.3 * 0.95
set in2_max 3.3 * 1.05
label in5 "+5V"
compute in5 @*(32/10), @/(32/10)
set in5_min 5.0 * 0.95
set in5_max 5.0 * 1.05
ignore in3
ignore in4
ignore in6
ignore in7
ignore in8

o MY /etc/fancontrol

INTERVAL=5
FCTEMPS=hwmon0/device/pwm4=hwmon0/device/temp2_input
FCFANS= hwmon0/device/pwm4=hwmon0/device/fan5_input
MINTEMP=hwmon0/device/pwm4=0
MAXTEMP=hwmon0/device/pwm4=45
MINSTART=hwmon0/device/pwm4=30
MINSTOP=hwmon0/device/pwm4=100

o MY /etc/rc.d/rc.fancontrol

#!/bin/sh
#
# /etc/rc.d/rc.fancontrol
#

fancontrol_start() {
/usr/local/sbin/fancontrol | logger &
}

fancontrol_stop() {
killall fancontrol
}

fancontrol_restart() {
fancontrol_stop
sleep 5
fancontrol_start
}


case "$1" in
'start')
fancontrol_start
;;
'stop')
fancontrol_stop
;;
'restart')
fancontrol_restart
;;
*)
echo "usage $0 start|stop|restart" ;;
esac

o My /etc/rc.d/rc.local

#!/bin/sh
#
# /etc/rc.d/rc.local:  Local system initialization script.
#
# Put any local startup commands in here.  Also, if you have
# anything that needs to be run at shutdown time you can
# make an /etc/rc.d/rc.local_shutdown script and put those
# commands in there.

# Why can't Linus respect the BIOS settings dammit!
/usr/bin/setleds +num
# ICH8R Software RAID Volumes
echo Mounting RAID Volumes
dmraid -ay
mount -v -t ntfs /dev/mapper/isw_fhbagdjeh_Twix1 /mnt/vista32
mount -v -t ntfs /dev/mapper/isw_fhbagdjeh_Twix2 /mnt/vista64
mount -v -t ntfs /dev/mapper/isw_fhbagdjeh_Twix3 /mnt/strider
mount -v -t ntfs /dev/mapper/isw_fhbagdjeh_Secure1 /mnt/secure
# This is to use the nv framebuffer for X rather than nVidia's
# lousy driver (plus it will remove the bootup penguins)
/usr/sbin/fbset -depth 32
# Harware monitoring stuff
/usr/local/bin/sensors -s
echo Starting Fan Control
if [ -x /etc/rc.d/rc.fancontrol ]; then
. /etc/rc.d/rc.fancontrol start
fi

o /etc/rc.d/rc.local_shutdown

#!/bin/sh
#
# /etc/rc.d/rc.local_shutdown:  Local system shutdown script.

if [ -x /etc/rc.d/rc.fancontrol ]; then
. /etc/rc.d/rc.fancontrol stop
fi

o And while I'm at it, my /etc/fstab...

/dev/sda4        /                xfs         defaults         1   1
/dev/sda1        /mnt/osx         hfsplus     rw               1   0
/dev/sda2        /mnt/winxp       ntfs        rw               1   0
/dev/sda3        /mnt/alfie       ntfs        rw               1   0
/dev/cdrom       /mnt/cdrom       auto        noauto,owner,ro  0   0
/dev/fd0         /mnt/floppy      auto        noauto,owner     0   0
devpts           /dev/pts         devpts      gid=5,mode=620   0   0
proc             /proc            proc        defaults         0   0

o ...as well as my ~/.profile

alias a=alias
a dir='ls -alFh'
a dim='ls -alFh | most'
a so='source ~/.profile'
a s='screen -D -R'

I think that's about it really

A farewell to LILO (One bootloader to rule them all, part 2)

2007-05-10T02:38:00.003+01:00

As mentioned previously, I set up a bunch of disks in a mixed RAID configuration on my new P5B Deluxe system, and merrily went on installing OS after OS.

The point of this exercise was to have my main OS (probably Vista 32) as well as a near half-a-terabyte partition set in RAID 0 mode, for speed, while some other stuff, for which disk performance is not that critical (OS-X, XP, Linux, more space), would reside quietly on a 3rd non raid disk. And while I'm at it, I would also setup a RAID 1 partition, as the Intel ICH8R "controller" of the P5B is now nice enough to allow you to create both a RAID0 and RAID1 drive out of the same 2 disks RAID array.

Which leaves us with the obvious problem of booting that lot, a capital problem indeed ( especially now that Vista is being such an ass about its boot loader). As an aside, I would like to point out that this is what happens when a company has a near monopoly with regards to installed OS base: They will obviously do their darnedest to prevent ANY other OS from eating even the smallest part of their market share. And I could comment on OS-X too. What is wrong with a standard partition scheme that you will recognize once you get your hand forced anyway (but not any sooner)? Why are you not happy with a big fat bootable primary partition with af type setup for you with Linux's fdisk?
And much much worse, how comes the partitions OS-X creates end up on the exact same block as the start of the next partition (have a look at your fdisk results in Linux - What the hell?)? I might talk about that later on, because this headache resulted in recently losing both a fresh install of XP along with a data partition and, what's worse, my Linux system. As the old proverb go, "Install XP, lose your data"...

So, considering that I have now done this operation twice, let me give you the long awaited heads up on the one bootloader to rule them all...

Accessing the ICH RAID disks under Linux

o Compile kernel with device mapper
o Pick up the latest devmapper package from ftp://sources.redhat.com/pub/dm/
o Pick and compile dmraid
o dmraid -ay (if dmraid complains about missing kernel options, add them!)
o dmraid -s (Yay!) or ls /dev/mapper

And while you're at it, you can add something similar to this at the end of your /etc/rc.d/rc.local file (Slackware)

echo mounting RAID Volumes
dmraid -ay
mount -v -t ntfs /dev/mapper/isw_fhbagdjeh_Twix1 /mnt/vista32
mount -v -t ntfs /dev/mapper/isw_fhbagdjeh_Twix2 /mnt/vista64
mount -v -t ntfs /dev/mapper/isw_fhbagdjeh_Twix3 /mnt/strider
mount -v -t ntfs /dev/mapper/isw_fhbagdjeh_Secure1 /mnt/secure

Becomes simple enough... when you've done that twice.

So long LILO, and thanks for all the flawless boots...

One word if you stubbornly try to get LILO to boot one of the RAID0 partition as I did: Agh!!!

Fatal: Sorry, don't know how to handle device 0xfd02 => compile with latest source, as it will compile with devmapper (device-mapper or LVM2) if library is present. Look for -DDEVMAPPER when compiling.
OK then, added devmapper option, make all, make install and...
"Fatal : device mapper : only linear boot device supported"

Enought with this. I'm sure the LILO devs will add it some day, but for now grub it is then.

Pub grub?

GNU's versions: WTF? Pick up 1.95 and... configure: error: LZO library version 1.02 or later is required
OK then, http://www.oberhumer.com/opensource/lzo/download/ to eventually realize: You should NOT grub 1.95. It's really grub 2.0 and too spanking new to be of any use. Instead, pick up grub 0.97 and live happily ever after!

Now, some of you might see some mention of a grub 0.97 dmraid patch, but because I'm actually using the 3rd non RAID disk (actually 1st one on my system) to install the bootloader, as well as ultimately boot, patching grub is unnecessary. I guess it would be necessary if we were to use our RAID0 as the boot disk. I'll have to try RAID5 one day, just for kicks :D

So, without further adieu to LILO, and with thanks to the Gentoo users:

grub --device-map=/dev/null
# My actual non-RAID boot disk
grub> device (hd0) /dev/sda
# The RAID0 array where the Vista's reside
grub> device (hd1) /dev/mapper/isw_fhbagdjeh_Twix
# Let's first add Linux, so that we can then edit the menu.lst
# My Linux is on /dev/sda4, hence hd0,3 with zero based indexes
grub> root (hd0,3)
Filesystem type is xfs, partition type 0x83
grub> setup (hd0,3)
Checking if "/boot/grub/stage1" exists... yes
Checking if "/boot/grub/stage2" exists... yes
Checking if "/boot/grub/xfs_stage1_5" exists... yes
Running "embed /boot/grub/xfs_stage1_5 (hd0,3)"... failed (this is not fatal)
Running "embed /boot/grub/xfs_stage1_5 (hd0,3)"... failed (this is not fatal)
Running "install /boot/grub/stage1 (hd0,3) /boot/grub/stage2 p /boot/grub/menu
.lst "... succeeded
Done.
grub> quit

At this stage, you wanna check your /boot/grub/device.map file. If you don't see the /dev/mapper entry for (hd1) as expected, you will have to add it manually.

From then on, you can go on creating your /boot/grub/menu.lst, which might eventually look something like this:

default 2
timeout 3

title  Slackware Linux 11.0
kernel (hd0,3)/boot/vmlinuz root=/dev/sda4

title  Slackware Linux 11.0 (Rescue)
kernel (hd0,3)/boot/vmlinuz.rescue root=/dev/sda4

title  Vista (32 bit)
rootnoverify (hd1,0)
chainloader +1

title  Vista (64 bit)
rootnoverify (hd1,1)
chainloader +1

title  Windows XP
rootnoverify (hd0,1)
chainloader +1

title  MacOS 10.4.8
rootnoverify (hd0,0)
chainloader +1

Parting words

And thus, I will be leaving with a "so long old Friend..." Since 1995, there haven't been much of a PC installed by yours truly which didn't display the unmistakable red greetings, and I had become such an expert at writing a /etc/lilo.conf, I could have done it blindfolded.
Still, I gotta give it to the GNU guys... Their solution to the booting problem is deceivingly elegant. I'm especially impressed with the "modify menu.lst on the fly" aspect of things and the fact that dmraid "just" works...

Yay! MacOS in 1680x1050 at last!

2007-03-30T00:49:00.001+01:00

As I mentioned earlier, I am using an MSI 7950 GT with 512 MB on my new rig, and 512 MB have been well known to be incompatible with Titan/Natit and others.
That is, until gotoh figured out that the issue was in the nVidia BIOS itself.
A simple mod/reflash is all that's needed now to get a 512 MB GeForce card to be detected successfully in JasOS. Well, actually, that's was the easy part. The rest, if you're not that familiar with OS-X might be a bit confusing (and threads of more than 25 pages sure do not help!).
Here's how I did proceed to get Natit installed and my card recognised (on an existing installation of Jas OS 10.4.8):

Flashed my Graphics Card following these steps
Downloaded the Natit_Dual_v0.2 package from this page (which got automatically extracted by Stuffit on the Desktop)
Copied over the content to /System/Library/Extensions with the command:
sudo cp -R Desktop/Natit.kext /System/Library/Extensions
Fired up "System Profiler" in "Utilities" (which you can access with Finder) and found that the Device ID for my Graphics Card was 0x0295
Edited Natit.kext/Contents/Info.plist and change the IOPCIMatch section string to 0x029510de...
in /System/Library/Extensions:
sudo chown -R root:wheel Natit*
in /System/Library
sudo rm -rf Extensions.mkext Extensions.kextcache
Rebooted and that was it!

UltraEdit 32 Explorer Integration in Vista

2007-03-28T23:09:00.001+01:00

Looks like Vista broke the integration of UltraEdit in the shell (even with latest version 13.00), without whom life is not worth living. Seriously, not being able to Edit/HexEdit ANY file with a right click really makes life unbearable.

Thankfully, other people have figured out how to do it, and it's really simple:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\*\shell\UltraEdit 32]
@="UltraEdit"

[HKEY_CLASSES_ROOT\*\shell\UltraEdit 32\command
@="\"C:\\Program Files\\IDM Computer Solutions\\UltraEdit-32\\uedit32.exe\" \"%1\""

(NB:If you create these keys manually, @ is the "(Default)" key in Regedit, and both keys are String values. Of course, quotes don't need to be escaped then)

The horror! The horror! (Hiding Vista partitions from one another)

2007-03-22T12:02:00.000Z

I'll cut to the chase. As you know, Vista's new bootloader is complate PITA, because it aims at taking control of all bootloading activities without asking (how rude!), and in typical Microsoft arrogance style, it thinks it can do a better job than you (how inconsiderate!).

Therefore, if you have any combination of XP, Vista, etc... that can be detected, you're pretty much screwed because Vista will insist on unifying the booting of all Windowses for you, which you might think is nice... until the day you remove the OS on which the bootloader is installed, and you realize that you can NO LONGER BOOT THE OTHERS. Don't believe me? Try that little experiment:
Install Vista. Then install another Vista on a different partition. The boot block for the second Vista will be written on the first partition. Now format the first Vista and do your darnedest to boot the second Vista. No way Jose: There's no boot block on that second partition and it will never boot. Well, if you don't mind me saying, that's a really gay way of operating a system.

So there's no way I'm gonna let that happen when I want to have a SINGLE bootloader under my COMPLETE CONTROL!

And it is a little know secret that, despite insisting on being ZE bootloader, Vista will happily receive the boot from another botoloader, as long as it has been installed on a machine where it thinks it is the only Windows OS, which is why it's a good advice to install Vista first, or unplug disks where you might have other Windowses installed. Unfortunately, that method of hiding Vista from Vista (I deem it likely that Vista is horrified from looking at itself in the mirror, hence the title) will not work in our case, where we want to have both Vista32 and Vista64 installed on the same disk. Oh, and you can try to change the partition type of the first Vista you installed, that won't work either. Windows is apparently able to detect itself regardless of the partition type We'll need much more powerful Maßnahmen gegen die Gewalt this time round.

Our solution then: Change the partition type AND fill the Windows partition bootblock with zeros. This is where we open our Swiss Army Knife of anything advanced: Linux.
Whatever you do, you should always install Linux before doing anything else (and be able to boot into it from a CD, floppy, whatever, even if the MBR is erased). Please chant with me now my brothers and sisters: "There is no salvation in the computing world but in Linux".

In my case, Linux is installed on HDD1 and the Vistas go to partition 1 & 2 of the RAID0 ICH8R array (HDD 2&3). You will need to have dmraid setup in Linux to access the ICH8R RAID partitions, but I'll post on that later. So the course of operations is:

Unplug HDD1 and install Vista32 on the first partition of the RAID0 unit. I even let Vista do my partitioning on this drive. Probably not the best idea (cfdisk seems to have problems with how Vista partitioned the drive, but fdisk is fine and see nice primary partitions there). I tend to use the Vista partitioner when I don't really care because in Vista, Megabytes are Megabytes, and not these lousy 1000^2 Megabites cfdisk and others insist on using. Thus, when I enter 45056 MB for the size, I do get a 44 GB partition exactly
While I'm at it, one piece of free advice. Don't reset your machine on Vista's first boot because you want to reinstall your bootloader or something. If Vista's first boot is interrupted, it will not be able to resume, so make sure that you can actually logon to Vista before you switch to doing something else.
Re-plug HDD1 and boot Linux. Fire up dmraid (dmraid -ay) to gain access to your RAID partitions.
Create a BACKUP of the Vista bootblock on RAID partition 1:
dd if=/dev/mapper/isw_fhbagdjeh_Twix1 of=/backup/vista32 count=16
Note 1: The reason I use 16 blocks (16 x 512 bytes) is because the BOOTSECT.BAK backup copy of the original drive bootblock done by Vista on installation is 8 KB big. If it's good enough for Vista, it's good enough for me
Note 2: The reason my RAID0 array is called Twix has everything to do with the French rebranding of the famous caramel bar from "Raider" to "Twix" a few years ago
Needless to say, if you ever lose that backup copy of your bootblock, you're screwed
Now we'll ERASE the backed up bootblock with:
dd if=/dev/zero of=/dev/mapper/isw_fhbagdjeh_Twix1 count=16
This makes sure that sneaky Windows will not be able to recognize this partition as one of its siblings
fire up fdisk (fdisk /dev/mapper/isw_fhbagdjeh_Twix) and unactivate the first partition, change its type to 'af' (how ironic, this is a MacOS partition type!) and activate the second partition (if the partition on which you plan to install Vista is not set as active/bootable, Vista will refuse to install!)
Unplug HDD1 and install Vista64. Vista will see the first partition alright, but not recognize it, so it will install a second bootloader on the second partition, which is what you want
Re-plug HDD1, boot Linux, issue the dmraid command, and restore the Vista32 boot record:
dd if=/backup/vista32 of=/dev/mapper/isw_fhbagdjeh_Twix1 count=16
Fire up fdisk again, and restore the first partition type to '7' (NTFS) and activate it (so that the 2 first partitions are marqued as bootable/active
Use the bootloader of your choice to boot these 2 Vistas independently from one another and be freed from Microsoft's Vista bootloader tie in. One more step into our "One bootloader to rule them all" brainwa^H^H^H^H^H^H^H program...

One bootloader to rule them all, part 1

2007-03-21T11:55:00.000Z

So, I bought myself an ASUS P5B Deluxe the other day, with all the good stuff that goes with it (Core 2 Duo E6400, 2 GB CAS 4 RAM, MSI GeForce 7950GX2 512 MB with passive cooling, etc.) and I also added 2 more of these relatively cheap Seagate 7200.10 ST3320620AS SATA II 298.1 GB drives (Hell will freeze over before I recognize that 1 GB is anything else but 1024^3 bytes!) to my existing one which brought my total to 3 of these units.

By the way, my tip to all Seagate 7200.10 drives owners: REMOVE THE HDD JUMPER IF YOU HAVE A SATA II MOTHERBOARD! With the jumper on, transfers are limited to SATA 1 speeds, which will effectively max the drive out to 1.5 Gbps. Without the jumper, you can reach the theoretical SATA II speed of 3.0 Gbps, so that's probably a wise thing to do... even if I doubt any hardware out there, and especially these disks, are able to max out SATA 1 speeds.

The Goal

Have OS-X (Jas OS 10.4.8 or later) plus Windows XP SP2 plus Vista 32 bit plus Vista 64 bit plus Linux (Slackware 11.0) as well as a hefty faaaaast partition installed in a mixed RAID (ICH8R Software RAID) configuration.

A lot of people will tell you that using the RAID0 ICHXR solution is stupid, that you will gain almost no performance, that it's too risky, and so on and so forth. Don't believe a word of it. I've been running a RAID0 array for my main OS and hefty 400 GB partition for the last 2 years, and I have been satisfied enough by this experience that I want to carry it over on my new machine. Preliminary tests show that I can effectively do a non RAID to RAID0 disk to disk copy at close to 70 MB/s (effective mean speed) on my new machine, while the earlier tests I remember with a non RAID0 destination were closer to 35 MB/s, so anybody who tells you that RAID0 doesn't bring anything is a fool.

How is this going to work in practice? Well, first of all, OSX will probably not be too happy on a RAID drive (won't know how to boot) and Linux is not that happy either. Sure, you can make it boot with dmraid and all, but as soon as your MBR is erased (and it WILL be erased), you won't have dmraid support on your boot CD so it's going to be a pain to reinstall. Yeah, I know you can use a bootdisk, and I even created my own custom Slackware boot/install CDs with custom kernel (that's how bender.rpc1.org was installed by the way), but I have become much lazier over the years, and I want to use the default Slackware boot CDs because it's convenient.

Therefeore, we're gonna keep on disk in AHCI (or so was the plan) to have OS-X, Linux, XP and other stuff for which we dont' care too much about perf, while we install the Vista's on the 2 other disks in RAID0 array.

"Good news, everyone!"

Now, here comes the pleasant surprise: Contrary to what was the case in the past, and which was one of the major reasons of my being pissed at these lousy software RAID implementations that could come close to Linux's md, it is NOW possible (at least with ICH8R on an ASUS P5B Deluxe) to create a mixed RAID configurations on the same disk array.
For instance, you can create BOTH a RAID0 unit for speed as well as a RAID1 unit for security with only 2 Disks. This really solves one of the major gripes I had with ICHR, because I want speed for most of my data (which is expandable), but I also want security for the limited set of my data which is not. In the past ICHR iterations, apart from buying 4 disks, you really didn't have a choice. With the latest one, you can have the cake and eat it too.

Now there is a limit to all good things (expect on Linux), and you can only create two raid units on the same array. But as long as I can have a big RAID0 drive and a small RAID1 drive on the same disks, I'm fine with that, so not a problem.

Here's how my current partitioning is done then:
o Disk 0 (non RAID, P5B SATA controller 0): 4 primaries, with OS-X as part 1 (40 GB), WinXP as part 2 (40 GB) and Linux as the last partition (~8 GB). The 3rd partition is used for data
o Disk 1 & 2 (on P5B SATA controllers 1 & 4): One 588 GB RAID0 unit/disk and one 4.1 GB RAID1 unit/disk. The RAID0 unit is partitioned as follows: 44 GB primary for Vista 32, 44 GB primary for Vista 64 and 500 GB for "Strider", my hefty faaaaast data drive (I would of course have preferred 512 GB for the latter, for binary beauty, but I don't think I want to go under 40 GB for a Vista installation.)

Booting all of these OSes will be the tricky part though, which I will detail in the upcoming "Farewell to LILO" post as well as probably part 2 of this post.

"Bad news, nobody..."

And now for the unpleasant surprise (can't have it all). I already had Linux, OS-X and XP installed on the first non RAID HDD (in AHCI mode), and even though I am now in RAID mode instead of AHCI in the P5B Deluxe BIOS, the HDD is in non RAID mode and should still be seen as AHCI (there are 3 modes on the P5B: IDE, RAID or AHCI). Except that intel, in their great wisdom, seem to have decided that, when in RAID mode, non RAID disks should be set as IDE or something, instead of AHCI, which means that WinXP won't boot (nothing that a good driver/reinstall won't fix) but more worrying, that OS-X is now unable to see its own drive as well...

Well, this is bad news because there is no patch for JaS OS 10.4.8 to be compatible with whatever mode ICH8R defines the drives as (and using JMicron as primary SATA controller is not really an option, regardless of whether it is compatible with JaS or not).

So I currently have to switch back from RAID to AHCI in the BIOS to be able to boot OS-X. Darn! The goal is to have "one bootloader to rule them all", not "one bootloader plus manual BIOS conf to rule them all"...

Sidenote

For those who might wonder why I didn't go for what would be considered as the more "sensible" approach of installing Vista(s) on a RAID1 partition, the answer, apart from the obvious hit in performance, is very simple: I tried just that, and then, after installing some updates, Windows refused to shut down for some reason, so I had to it reset. Once I rebooted, the RAID bios detected that my RAID1 array was out of sync and that it needed to be rebuilt. But when I did that in Windows, using the Intel Matrix Storage tools, I realized that this would take forever (about 1% per 30 secs or so on a 44 GB drive). Considering how stable Vista is on shutdown/reboot operations (I'm not even talking of sleep, which, as everybody knows, is simply dreadful and does not work), and how often I am likely to hit the reset button to show that OS who is the master, I'm not going to waste my life in degraded performance mode, having to wait for my array to be rebuilt. This is only the reason why I went for a small RAID1 partition by all standards. I can afford a couple minutes to rebuild a 4 GB drive, but 1 hour for a 40+ one, no way!

You might remember me...

2007-03-03T02:52:00.001Z

Hi, I'm hacker >NIL: ("redirect 'slash' nil").

You might remember me for such hacks as "Pioneer DVR-108 v1.18 RPC-1 + 12xRip + nx4all" or "The Matshita UJDA autopatcher", and events such as "Why the hell is rpc1.org down again?!?"...

On this not-so-regularly updated blog, I purport to present you with the unglamorous vicissitudes of managing to make computers, and all things computer related, achieve what you want.

And so, without further ado ...